Merge branch 'master' into win32color

Conflicts:
	src/options.c

.gitignore

@@ -6,6 +6,7 @@
 .DS_Store
 aclocal.m4
 ag
+ag.exe
 autom4te.cache
 cachegrind.out.*
 callgrind.out.*
@@ -27,4 +28,4 @@ stamp-h1
 tests/*.err
 tests/big/*.err
 tests/big/big_file.txt
-the_silver_searcher.spec
+the_silver_searcher.spec

CONTRIBUTING.md

@@ -0,0 +1,16 @@
+## Contributing
+
+I like when people send pull requests. It validates my existence. If you want to help out, check the [issue list](https://github.com/ggreer/the_silver_searcher/issues?sort=updated&state=open) or search the codebase for `TODO`. Don't worry if you lack experience writing C. If I think a pull request isn't ready to be merged, I'll give feedback in comments. Once everything looks good, I'll comment on your pull request with a cool animated gif and hit the merge button.
+
+### Running the test suite
+
+If you contribute, you might want to run the test suite before and after writing
+some code, just to make sure you did not break anything. Adding tests along with
+your code is nice to have, because it makes regressions less likely to happen.
+Also, if you think you have found a bug, contributing a failing test case is a
+good way of making your point and adding value at the same time.
+
+The test suite uses [Cram](https://bitheap.org/cram/). You'll need to build ag
+first, and then you can run the suite from the root of the repository :
+
+    make test

Makefile.am

@@ -11,10 +11,10 @@ dist_bashcomp_DATA = ag.bashcomp.sh
 
 EXTRA_DIST = Makefile.w32 LICENSE NOTICE the_silver_searcher.spec README.md
 
-test:
+test: ag
 	cram -v tests/*.t
 
-test_big:
+test_big: ag
 	cram -v tests/big/*.t
 
 .PHONY : all test clean

Makefile.w32

@@ -1,4 +1,5 @@
-VERSION=$(shell grep -Po "(?<=\[)([0-9.]+.[0-9]+.[0-9]+)(?=\])" configure.ac)
+GREP=grep
+VERSION:=$(shell $(GREP) -Po "(?<=\[)([0-9.]+.[0-9]+.[0-9]+)(?=\])" configure.ac)
 
 CC=gcc
 

README.md

@@ -1,39 +1,44 @@
-# The Silver Searcher #
+# The Silver Searcher
 
 A code searching tool similar to `ack`, with a focus on speed.
 
 [![Build Status](https://travis-ci.org/ggreer/the_silver_searcher.svg?branch=master)](https://travis-ci.org/ggreer/the_silver_searcher)
 
-<a href="https://floobits.com/ggreer/ag/redirect">
-  <img alt="Floobits status" width="100" height="40" src="https://floobits.com/ggreer/ag.png" />
-</a>
+[![Floobits Status](https://floobits.com/ggreer/ag.png)](https://floobits.com/ggreer/ag/redirect)
 
+Do you know C? I invite you to pair with me to [help me get to Ag 1.0](http://geoff.greer.fm/2014/10/13/help-me-get-to-ag-10/).
 
-## What's so great about Ag? ##
 
-* It searches code about 3–5× faster than `ack`.
+## What's so great about Ag?
+
+* It is an order of magnitude faster than `ack`.
 * It ignores file patterns from your `.gitignore` and `.hgignore`.
-* If there are files in your source repo you don't want to search, just add their patterns to a `.agignore` file. \*cough\* extern \*cough\*
+* If there are files in your source repo you don't want to search, just add their patterns to a `.agignore` file. (\*cough\* extern \*cough\*)
 * The command name is 33% shorter than `ack`, and all keys are on the home row!
 
+Ag is quite stable now. Most changes are new features, minor bug fixes, or performance improvements. It's much faster than Ack in my benchmarks:
 
-## How is it so fast? ##
+    ack test_blah ~/code/  104.66s user 4.82s system 99% cpu 1:50.03 total
 
-* Searching for literals (no regex) uses [Boyer-Moore-Horspool strstr](http://en.wikipedia.org/wiki/Boyer%E2%80%93Moore%E2%80%93Horspool_algorithm).
-* Files are `mmap()`ed instead of read into a buffer.
-* If you're building with PCRE 8.21 or greater, regex searches use [the JIT compiler](http://sljit.sourceforge.net/pcre.html).
-* Ag calls `pcre_study()` before executing the regex on a jillion files.
-* Instead of calling `fnmatch()` on every pattern in your ignore files, non-regex patterns are loaded into an array and binary searched.
-* Ag uses [Pthreads](http://en.wikipedia.org/wiki/POSIX_Threads) to take advantage of multiple CPU cores and search files in parallel.
+    ag test_blah ~/code/  4.67s user 4.58s system 286% cpu 3.227 total
 
-I've written several blog posts showing how I've improved performance. These include how I [added pthreads](http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-pthreads/), [wrote my own `scandir()`](http://geoff.greer.fm/2012/09/03/profiling-ag-writing-my-own-scandir/), [benchmarked every revision to find performance regressions](http://geoff.greer.fm/2012/08/25/the-silver-searcher-benchmarking-revisions/), and profiled with [gprof](http://geoff.greer.fm/2012/02/08/profiling-with-gprof/) and [Valgrind](http://geoff.greer.fm/2012/01/23/making-programs-faster-profiling/).
+Ack and Ag found the same results, but Ag was 34x faster (3.2 seconds vs 110 seconds). My `~/code` directory is about 8GB. Thanks to git/hg/svn-ignore, Ag only searched 700MB of that.
+
+There are also [graphs of performance across releases](http://geoff.greer.fm/ag/speed/).
 
+## How is it so fast?
 
-## Installation ##
+* Ag uses [Pthreads](https://en.wikipedia.org/wiki/POSIX_Threads) to take advantage of multiple CPU cores and search files in parallel.
+* Files are `mmap()`ed instead of read into a buffer.
+* Literal string searching uses [Boyer-Moore strstr](https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_search_algorithm).
+* Regex searching uses [PCRE's JIT compiler](http://sljit.sourceforge.net/pcre.html) (if Ag is built with PCRE >=8.21).
+* Ag calls `pcre_study()` before executing the same regex on every file.
+* Instead of calling `fnmatch()` on every pattern in your ignore files, non-regex patterns are loaded into arrays and binary searched.
+
+I've written several blog posts showing how I've improved performance. These include how I [added pthreads](http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-pthreads/), [wrote my own `scandir()`](http://geoff.greer.fm/2012/09/03/profiling-ag-writing-my-own-scandir/), [benchmarked every revision to find performance regressions](http://geoff.greer.fm/2012/08/25/the-silver-searcher-benchmarking-revisions/), and profiled with [gprof](http://geoff.greer.fm/2012/02/08/profiling-with-gprof/) and [Valgrind](http://geoff.greer.fm/2012/01/23/making-programs-faster-profiling/).
 
-### Gentoo
 
-    emerge the_silver_searcher
+## Installing
 
 ### OS X
 
@@ -43,46 +48,54 @@ or
 
     port install the_silver_searcher
 
-### Arch Linux
 
-    pacman -S the_silver_searcher
+### Linux
 
-### Debian unstable
+* Ubuntu >= 13.10 (Saucy) or Debian >= 8 (Jessie)
 
-    apt-get install silversearcher-ag
+        apt-get install silversearcher-ag
+* Fedora 19+
 
-### Ubuntu 13.10 or later
+        yum install the_silver_searcher
+* RHEL7+
 
-    apt-get install silversearcher-ag
+        rpm -Uvh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
+        yum install the_silver_searcher
+* Gentoo
 
-### FreeBSD
+        emerge the_silver_searcher
+* Arch
 
-    pkg install the_silver_searcher
+        pacman -S the_silver_searcher
 
-or
+* Slackware
 
-    pkg_add -r the_silver_searcher
+        sbopkg -i the_silver_searcher
 
-To build from source on FreeBSD:
 
-    make -C /usr/ports/textproc/the_silver_searcher install clean
+### BSD
 
-### OpenBSD
+* FreeBSD
 
-    pkg_add the_silver_searcher
+        pkg install the_silver_searcher
+* OpenBSD/NetBSD
 
-To build from source on OpenBSD:
+        pkg_add the_silver_searcher
 
-    cd /usr/ports/textproc/the_silver_searcher && make install
 
 
-If you want a CentOS rpm or Ubuntu deb, take a look at [Vikram Dighe's packages](http://swiftsignal.com/packages/).
+## Building from source
 
-
-## Building from source ##
+### Building master
 
 1. Install dependencies (Automake, pkg-config, PCRE, LZMA):
-    * Ubuntu:
+    * OS X:
+
+            brew install automake pkg-config pcre xz
+        or
+
+            port install automake pkgconfig pcre xz
+    * Ubuntu/Debian:
 
             apt-get install -y automake pkg-config libpcre3-dev zlib1g-dev liblzma-dev
     * Fedora:
@@ -92,12 +105,6 @@ If you want a CentOS rpm or Ubuntu deb, take a look at [Vikram Dighe's packages]
 
             yum -y groupinstall "Development Tools"
             yum -y install pcre-devel xz-devel
-    * OS X:
-
-            brew install automake pkg-config pcre
-        or
-
-            port install automake pkgconfig pcre
     * Windows: It's complicated. See [this wiki page](https://github.com/ggreer/the_silver_searcher/wiki/Windows).
 2. Run the build script (which just runs aclocal, automake, etc):
 
@@ -111,62 +118,48 @@ If you want a CentOS rpm or Ubuntu deb, take a look at [Vikram Dighe's packages]
         sudo make install
 
 
+### Building a release tarball
 
-## Current development status ##
+Building release tarballs requires the same dependencies, except for automake and pkg-config. Once you've installed the dependencies, just run:
 
-It's quite stable now. Most changes are new features, minor bug fixes, or performance improvements. It's much faster than Ack in my benchmarks.
+    ./configure
+    make
+    make install
 
-    ack blahblahblah ~/code  6.59s user 1.94s system 99% cpu 8.547 total
+You may need to use `sudo` or run as root for the make install.
 
-    ag blahblahblah ~/code  1.39s user 1.81s system 229% cpu 1.396 total
 
+## Editor Integration
 
-## Editor Integration ##
+### Vim
 
-### TextMate ###
+You can use Ag with [ack.vim][] by adding the following line to your `.vimrc`:
 
-TextMate users can use Ag with [my fork](https://github.com/ggreer/AckMate) of the popular AckMate plugin, which lets you use both Ack and Ag for searching. If you already have AckMate you just want to replace Ack with Ag, move or delete `"~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack"` and run `ln -s /usr/local/bin/ag "~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack"`
+    let g:ackprg = 'ag --nogroup --nocolor --column'
 
-### Vim ###
+or:
 
-You can use Ag with [ack.vim][] by adding the following line to your `.vimrc`:
+    let g:ackprg = 'ag --vimgrep'
 
-    let g:ackprg = 'ag --nogroup --nocolor --column'
+Which has the same effect but will report every match on the line.
 
 There's also a fork of ack.vim tailored for use with Ag: [ag.vim][]
 [ack.vim]: https://github.com/mileszs/ack.vim
 [ag.vim]: https://github.com/rking/ag.vim
 
-### Emacs ###
+### Emacs
 
-You can use use [ag.el][] as an Emacs fronted to Ag.
+You can use [ag.el][] as an Emacs fronted to Ag.
 
 [ag.el]: https://github.com/Wilfred/ag.el
 
+### TextMate
 
-## Contributing ##
-
-I like when people send pull requests. It validates my existence. If you want to help out, check the [issue list](https://github.com/ggreer/the_silver_searcher/issues?sort=updated&state=open) or search the codebase for `TODO`. Don't worry if you lack experience writing C. If I think a pull request isn't ready to be merged, I'll give feedback in comments. Once everything looks good, I'll comment on your pull request with a cool animated gif and hit the merge button.
-
-
-## TODO ##
-
-A special thanks goes out to Alex Davies. He has given me some excellent recommendations to improve Ag. Many of these things are still on my list:
-
-* Optimizations
-  * Write a benchmarking script that tweaks various settings to find what's fastest.
-* Features
-  * Behave better when matching in files with really long lines.
-  * Report "match found at position X of line N" if line is > 10k chars.
-* Windows support
-  * `readdir()` and `stat()` are much slower on Windows. Use `FindNextFile()` instead.
-  * Support Visual Studio instead of autotools?
-  * Need to use pthreads-win32 or something similar.
-
+TextMate users can use Ag with [my fork](https://github.com/ggreer/AckMate) of the popular AckMate plugin, which lets you use both Ack and Ag for searching. If you already have AckMate you just want to replace Ack with Ag, move or delete `"~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack"` and run `ln -s /usr/local/bin/ag "~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack"`
 
-## Other stuff you might like ##
+## Other stuff you might like
 
-* [Ack](https://github.com/petdance/ack) - Better than grep. Without Ack, Ag would not exist.
+* [Ack](https://github.com/petdance/ack2) - Better than grep. Without Ack, Ag would not exist.
 * [AckMate](https://github.com/protocool/AckMate) - An ack-powered replacement for TextMate's slow built-in search.
 * [ack.vim](https://github.com/mileszs/ack.vim)
 * [ag.vim]( https://github.com/rking/ag.vim)

ag.bashcomp.sh

@@ -30,6 +30,7 @@ _ag() {
     --file-search-regex
     --files-with-matches
     --files-without-matches
+    --fixed-strings
     --follow
     --group
     --nogroup
@@ -62,20 +63,23 @@ _ag() {
     --stats
     --unrestricted
     --version
+    --vimgrep
     --word-regexp
     --workers
   '
   shtopt='
     -a -A -B -C -D
-    -f -g -G -h -i
-    -l -L -m -n -p
-    -Q -r -R -s -S
-    -t -u -U -v -V
-    -w -z
+    -f -F -g -G -h
+    -i -l -L -m -n
+    -p -Q -r -R -s
+    -S -t -u -U -v
+    -V -w -z
   '
 
+  types=$(ag --list-file-types |grep -- '--')
+
   # these options require an argument
-  if [[ "${prev}" == -@(A|B|C|G|g|m) ]] ; then
+  if [[ "${prev}" == -[ABCGgm] ]] ; then
     return 0
   fi
 
@@ -100,13 +104,8 @@ _ag() {
 
   case "${cur}" in
     -*)
-          if [[ "${COMP_CWORD}" -eq 1 ]] ; then
-            COMPREPLY=( $(compgen -W \
-              "${lngopt} ${shtopt}" -- "${cur}") )
-          else
-            COMPREPLY=( $(compgen -W \
-              "${lngopt} ${shtopt}" -- "${cur}") )
-          fi
+          COMPREPLY=( $(compgen -W \
+            "${lngopt} ${shtopt} ${types}" -- "${cur}") )
           return 0;;
     *)
           _filedir

configure.ac

@@ -1,6 +1,6 @@
 AC_INIT(
     [the_silver_searcher],
-    [0.24.1],
+    [0.28.0],
     [https://github.com/ggreer/the_silver_searcher/issues],
     [the_silver_searcher],
     [https://github.com/ggreer/the_silver_searcher])
@@ -18,9 +18,10 @@ m4_ifdef(
 PKG_CHECK_MODULES([PCRE], [libpcre])
 
 m4_include([m4/ax_pthread.m4])
-AX_PTHREAD([
-    AC_CHECK_HEADERS([pthread.h])
-])
+AX_PTHREAD(
+    [AC_CHECK_HEADERS([pthread.h])],
+    [AC_MSG_WARN([No pthread support. Ag will be slower due to running single-threaded.])]
+)
 
 # Run CFLAGS="-pg" ./configure if you want debug symbols
 CFLAGS="$CFLAGS $PTHREAD_CFLAGS $PCRE_CFLAGS -Wall -Wextra -Wformat=2 -Wno-format-nonliteral -Wshadow -Wpointer-arith -Wcast-qual -Wmissing-prototypes -Wno-missing-braces -std=gnu89 -D_GNU_SOURCE -O2"
@@ -46,9 +47,11 @@ AS_IF([test "x$enable_lzma" != "xno"], [
 
 AC_CHECK_DECL([PCRE_CONFIG_JIT], [AC_DEFINE([USE_PCRE_JIT], [], [Use PCRE JIT])], [], [#include <pcre.h>])
 
+AC_CHECK_DECL([CPU_ZERO, CPU_SET], [AC_DEFINE([USE_CPU_SET], [], [Use CPU_SET macros])] , [], [#include <sched.h>])
+
 AC_CHECK_MEMBER([struct dirent.d_type], [AC_DEFINE([HAVE_DIRENT_DTYPE], [], [Have dirent struct member d_type])], [], [[#include <dirent.h>]])
 
-AC_CHECK_FUNCS(fgetln getline realpath strlcpy strndup vasprintf madvise posix_fadvise)
+AC_CHECK_FUNCS(fgetln getline realpath strlcpy strndup vasprintf madvise posix_fadvise pthread_setaffinity_np)
 
 AC_CONFIG_FILES([Makefile the_silver_searcher.spec])
 AC_CONFIG_HEADERS([src/config.h])