text/scanner: SkipComments is ignored without ScanComments

[c4rlo]

Go version

go1.23.4

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/carlo/.cache/go-build'
GOENV='/home/carlo/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/carlo/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/carlo/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.4'
GODEBUG=''
GOTELEMETRY='off'
GOTELEMETRYDIR='/home/carlo/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v3'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/carlo/src/go-scanner-bug/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build186288891=/tmp/go-build -gno-record-gcc-switches'

What did you do?

The text/scanner docs say:

Predefined mode bits to control recognition of tokens. For instance, to configure a Scanner such that it only recognizes (Go) identifiers, integers, and skips comments, set the Scanner's Mode field to:

ScanIdents | ScanInts | SkipComments

With the exceptions of comments, which are skipped if SkipComments is set, unrecognized tokens are not ignored.

However, it seems that SkipComments only works as advertised when combined with ScanComments, despite the example clearly implying otherwise.

The following code demonstrates (see go.dev.play link):

package main

import (
	"fmt"
	"strings"
	"text/scanner"
)

func main() {
	testScanner(scanner.ScanIdents|scanner.ScanInts|scanner.SkipComments, "// comment")
	testScanner(scanner.ScanIdents|scanner.ScanInts|scanner.ScanComments|scanner.SkipComments, "// comment")
}

func testScanner(mode uint, input string) {
	var sc scanner.Scanner
	sc.Init(strings.NewReader(input))
	sc.Mode = mode
	for sc.Peek() != scanner.EOF {
		tok := sc.Scan()
		fmt.Printf("[%s:'%s'] ", scanner.TokenString(tok), sc.TokenText())
	}
	fmt.Println()
}

What did you see happen?

The output is:

["/":'/'] ["/":'/'] [Ident:'comment'] 
[EOF:''] 

What did you expect to see?

I would expect the output to be:

[EOF:''] 
[EOF:''] 

[ianlancetaylor]

CC @griesemer

[gabyhelp]

Related Issues

• go/scanner: bug? if scan.Init with Mode(0) #10213 (closed)
• go/scanner: Scanner.Scan sometimes returns token.ILLEGAL and empty literal string #28112 (closed)

Related Code Changes

• cmd/compile/internal/syntax: implement comment reporting in scanner

Related Documentation

• Package scanner > Constants

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

I think this is just documentation, the definition of SkipComments notes the requirement to also set ScanComments

SkipComments = 1 << -skipComment // if set with ScanComments, comments become white space