TODO

TODO:
==========================

Document use of bash anon named pipes with @ syntax
spaces in various places in grammar (between @ and filename, spaces inside of [] and () for calls)

xor hangs with large keyfiles
Context objects, Classes method_missing delegate to singleton instance
TTY handling: default trailing newline, colors/escapes for non-printable
tr, replace functions
interleave/concat multiple streams
vararg functions
switch endianness

fix
-----------
bug in b64 decoding of HFF= => HFE=

robustness
-----------
* graceful handling of mismatched data size (key size, base64 chunks, xor)
* errors for bad function name or args
* case insensitivity for ~b16
* high level classification of functions (util, bitwise, hash, cipher, mac, etc)
* handling of optional arguments (IVs/nonces, # rounds, etc) without breaking pipe argument prepending
* escape/encode non-printable chars if outputting to tty http://blog.bogojoker.com/2009/04/check-if-your-ruby-script-is-in-a-pipe/
* description for functions in help

completeness
-----------
base64 variants
* %/url, b32, ascii85, escape-sequence (\x, String.dump), xml/html
* bitwise ops: variants for mismatched lengths?


features
---------------
functions
  bitwise ops
    shift right/left (circular rotate vs fill)
    not/and/or (similar to xor)
  hashes  
    crc/adler
    crypt/scrypt/bcrypt/pbkdf
  ciphers
    substitution: caeser/rot/dvorak/vigenère/affine/atbash/avgad/albam
    physical: rotor/enigma
  byte ops
    prepend/append
    indexed head/tail/substring
    reverse bits/nibbles/bytes	
    repeat bytes
  compression
    gzip/deflate/lz*/bzip2/snappy/rle
      http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv
  generators
    random bytes

revamp streams
  http://readruby.io/io
  https://github.com/javanthropus/io-like
  https://github.com/javanthropus/stream
  https://github.com/rubygems/rubygems/blob/master/lib/rubygems/package/digest_io.rb
  https://github.com/krypt/binyo
  https://gist.github.com/tlrobinson/967710
  https://www.ruby-forum.com/topic/58563
  https://github.com/jdleesmiller/event_state
  https://github.com/slagyr/statemachine/
namespacing in case of simple name collisions
string literals
  alternative quotes (a la ruby/perl/python, %q[] %Q{} qq() """ ''') http://en.wikibooks.org/wiki/Ruby_Programming/Alternate_quotes
  allow \xXX escaping
extension mechanism (via other gems or simple files)
varargs for functions
non pipe statements variable writing/reading


random ideas
-------------
`strings` extraction
regex search/replace support
fn for executing shell/ruby
multi-round hash support


cleanliness
-----------
refactor variable buffering (probably will improve perf)
break up functions

goals and philosophy
==========================
functions on streams of bytes
kiss: crypto, hashes, bitwise ops, encodings, compression, simple transforms
most complex use case: generate rails cookie from marshalled data and secret
strive for streaming impls and low memory usage even for large input documents
pull/read (vs push/write) based API
shell friendly syntax
maximum portability, no native gem deps
extensible via external gems adding functions



scratch notes
==========================


reverse(input,chunksize=1,boundarysize=nil)
cat(input1,input2,...,inputn)

ruby -e 'puts ["c","s"].product(["i","y"]).product(["ph","f"]).product(["r","er","yr","ir"]).map{|a| a.flatten.join}'
ciphr
cipher
ciphyr
ciphir
cifr
cifer
cifyr
cifir
cyphr
cypher
cyphyr
cyphir
cyfr
cyfer
cyfyr
cyfir
siphr
sipher
siphyr
siphir
sifr
sifer
sifyr
sifir
syphr
sypher
syphyr
syphir
syfr
syfer
syfyr
syfir




Gem::Specification.find_all.to_a.select{|s| s.dependencies.any?{|d| d.name=='slop'}}.map{|s| s.name}
https://codeclimate.com/github/cloudfoundry/vmc/VMC::Plugin/source_listing.js?line_end=53&line_start=11&path=lib%2Fvmc%2Fplugin.rb
https://github.com/mmozuras/pronto
http://stackoverflow.com/questions/11827950/create-an-extensible-ruby-gem
https://bitbucket.org/ged/strelka/src/dc4b6998813161ab4c94656ce144aa36a85ab65e/lib/strelka/discovery.rb




http://weblog.rubyonrails.org/2009/9/1/gem-packaging-best-practices/
https://gist.github.com/rtomayko/54177




11111111 | 11111111 | 11111111
111 | 111 | 111 | 111 | 111 | 111 | 111 | 111 

1 2 3 4 5 6 7 8*1 2 3 4 5 6 7 8*1 2 3 4 5 6 7 8
1 2 3*1 2 3*1 2 3*1 2 3*1 2 3*1 2 3*1 2 3*1 2 3


~b64,hmac(abcd,~b64),b16,xor(deef,~b64)
~des

=abcd/^hex/b64/hmac[=abcd/^b64]=/

{abcd},~hex,b64/hmac-sha1[[{abcd},b64]]

=abcd ~hex hmac[=foo] ?key

~[],?@^#{}


0x01



foo#bar^baz[]
foo;bar|baz()

(foo|bar)|baz

((foo)|bar)|baz

assign var:		?varname
ref var:		#varname
literal:		=value
separator:		" "
file reference:	@filepath

% in url
used in b64: 	_-:!/.+=
shell ok: 		~,.-_{}[]/?%#@^+=:
shell not ok: 	()|!$&*\;'"<>` 

	sh 	b64
`		x
~	x 	x
!	
@	x 	x
#	x 	x
$	 	x
%	x 	x
^	x 	x
&	 	x
*	 	x
()	 	x
-	x 	
_	x 	
=	x 	b
+	x 	
[]	x 	x
{}	x 	x
;	 	x
:	x 	
'		x
"	 	x
,	x 	x
.	x 	
/	x 
<>	 	x
?	x 	x



x(a)|y(b)|z(c)



					z
			
			y				c
	
		x		b

	a






         file
           |
           V
         ~hex
           |
           V
stdin --> aes --> hex --> stdout


         file1   file2
           |       |
           V       V
         ~hex    ~hex
           |       |
           V       V
stdin --> aes --> aes -> hex --> stdout


stdin --> md5 --> hex --> stdout


stdin --> md5 --> md5 --> ... -> md5 --> hex --> stdout




base64: read multiple loop
  input.read(4) {|bytes,eof|
    output.write(encode(bytes))
  }

cipher: read full init, read arbitrary loop, special end
  key.read() {|bytes,eof|
    @bytes = bytes
  }
  input.read(256) {|bytes,eof|
    output.write(cipher.update)
  }

xor-repeat: read full init, read arbitrary loop
  key.read() {|bytes,eof|}


xor-trunc: read arbitrary loop
  key.read(1) && input.read(1) {

  }


md5: read arbitrary loop, special end

hmac: read full init, read arbitrary loop, special end

def pump
  if input.size >= 1 && key.size >= 1
    output.write(input.read(1) ^ key.read(1))
  end
end



IDEAS:

method_missing for arguments in function body
  

+ for concat
* for repeat
^ for recurse