Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds capability for colored scatter plots (issue #195) #396

Merged
merged 25 commits into from
Nov 24, 2017
Merged

Adds capability for colored scatter plots (issue #195) #396

merged 25 commits into from
Nov 24, 2017

Conversation

Scorpiokat
Copy link
Contributor

@Scorpiokat Scorpiokat commented Oct 7, 2017

I have closed the previous pull request, and created this one according to your comments @kortschak and @ctessum. Thanks!

@ctessum I decided to send this PR anyway (and it's up to you, obviously, to accept it or not right now), so that at least the plot has this capability and it will fix the #195 issue. Meanwhile, I will start working on the alternative solution which you have proposed. It should definitely look much more elegant and be an improvement of the existing way of making scatter, scatter colour and bubble plots.

@Scorpiokat
Copy link
Contributor Author

@ctessum Chris, I am a bit confused. If we decide not to merge this PR, and follow the directions from your comments here, may I ask 2 things that I do not clearly understand:

  1. It looks like we are trying to avoid using Z as a third parameter, but then here c, err := cm.At(scatterData.Z(i)) it suddenly appears. If we add Z to scatter.go this will cause a need to add it in a lot of other places like line.go too. If that is fine, I will do so.
  2. What is the role of i here s.GlyphStyleFunc = func(i int) draw.GlyphStyle and where it comes from? We will use it to plot scatter color or bubble (or any other) plots?

@ctessum
Copy link
Contributor

ctessum commented Oct 16, 2017

Hi @Scorpiokat,

In Go, functions can be variables.

How the concept I proposed would work is that we would add a field to the Scatter struct that is a function (GlyphStyleFunc). This is different than adding a method to Scatter, because method functions are defined at compile time-but field functions can be specified at run-time by the user.

Because it is a field and not a method, by default GlyphStyleFunc is nil. However, a user could at their discretion set GlyphStyleFunc to a function of their choosing, which would override the default style (that does not vary among scatter points) with the style specified by GlyphStyleFunc (which can vary among scatter points).

There is already something similar implemented in the Sankey plotter. In this example you can see how we can specify custom style functions (FlowStyle and StockStyle). In the Sankey plotter logic, you can see that here we specify default StockStyle and FlowStyle fields (which can be changed by the user before plotting), and then here we call the function variable to determine the style of each item.

So, to answer your specific questions:

  1. Z would not be used in the code of the plotter itself, but could necessary in some user-specified GlyphStyleFuncs. In those cases, it would be up to the user to provide a values for Z or whatever other variable name they choose to call it.
  2. i refers to the point index number, so for the first point i=0, for the second point i=1, etc. It is necessary to have i as a function argument so that users can create functions that give different styles for different points (e.g., a user could create a function that colors the first point red, the second point blue, etc.)

Let me know whether that makes sense.

@kortschak
Copy link
Member

You may also like to see how bíogo rings does this kind of thing where data series are reflected on to see if they want to do styling. Note that, from memory, @eaburns has indicated that he does not really want to see things like this in the plotters here.

@Scorpiokat
Copy link
Contributor Author

@ctessum thanks you for your comments.

I got the idea, but still not completely clear on the Z (or whichever variable that is), and how all this c, err := cm.At(scatterData.Z(i)) works, sorry. scatterData.Z(i) should be a function that depends on i and results in float64? That is a bit confusing.

And will cm.At give the same nice 'colour intensity growing' effect like palette.Heat? Or it will depend on the float64 in the brackets?

@ctessum
Copy link
Contributor

ctessum commented Oct 18, 2017

What I didn't include in the original explanation was that I was asssuming that scatterData would be some type that had a method Z(i int) float64 that returned the z value at index i. It wouldn't need to be like that, for example you could also have var z []float64, and then it would be c, err := cm.At(z[i]). Regardless, the plotter code itself wouldn't need to know anything about z values, it would just be in the example showing how to make colored scatter plots.

@ctessum
Copy link
Contributor

ctessum commented Oct 18, 2017

An independent question is whether there is consensus regarding whether this is a good idea. This may not be the correct location for that discussion, however.

@Scorpiokat
Copy link
Contributor Author

Scorpiokat commented Oct 19, 2017

@ctessum I have sent a new commit with the changes you have proposed. As this is a 2-in-1 PR, it became a bit messy.
I will delete scatterColor.go, scatterColor_test.go and the plot, if you prefer the second option. Please let me know.

Copy link
Member

@sbinet sbinet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a first pass.

I would remove the scatterDataNew data set from the ExampleScatter as it clutters the output plot somewhat.

I would also completely remove the ScatterColor type which isn't needed anymore.

@@ -10,10 +10,14 @@ import (
"math/rand"
"testing"

"fmt"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put "fmt" and "os" as part of the stdlib import block

dc := draw.New(img)
dc = draw.Crop(dc, 0, -legendWidth, 0, 0) // Make space for the legend.
p.Draw(dc)
w, err := os.Create("testdata/scatter.png")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are missing a defer w.Close() here as well as an explicit err = w.Close() (with the according error check) after the call to png.WriteTo(w).

package plotter

import (
"gonum.org/v1/plot/palette"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this gonum.org/v1/plot/palette import should be part of the gonum.org import block below.

"math/rand"
"testing"

"fmt"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and fmt and os should be part of the stdlib import block above.

dc := draw.New(img)
dc = draw.Crop(dc, 0, -legendWidth, 0, 0) // Make space for the legend.
p.Draw(dc)
w, err := os.Create("testdata/scattercolor.png")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments than above about defer w.Close() and the explicit w.Close().

"os"
)

// ExampleScatter draws some scatter coloured points.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/ExampleScatter/ExampleScatter_color/

)

// ExampleScatter draws some scatter coloured points.
func ExampleScatterColor() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/ExampleScatter/ExampleScatter_color/


pal := palette.Heat(12, 1)

sc, err := NewScatterColor(scatterColorData, pal)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use NewScatter instead.

// determines the colour of a scatter plot point.
// Scatter implements the Plotter interface, drawing
// a glyph for each of a set of points.
type ScatterColor struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this type. it's not needed anymore.

@Scorpiokat
Copy link
Contributor Author

Scorpiokat commented Oct 27, 2017

Thank you @sbinet . I decided to delete scattercolor.go and scattercolor_test.go completely, as they are excessive to this implementation.

I am not very happy about the look of the legend on the right though, but have no idea on a better way to locate it.

@sbinet
Copy link
Member

sbinet commented Oct 27, 2017

I personnally prefered keeping the original scatter example untouched and add a new one exercizing the new scatter-color plot.

(but I'll let others chime in)


s.GlyphStyle = DefaultGlyphStyle

s.GlyphStyleFunc = func(int) draw.GlyphStyle {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could keep it nil, couldn't we? (as we handle the nil case in the Scatter.Plot method)

actually, I think we could leave the whole NewScatter function untouched.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my comment above is still relevant, I believe.

Copy link
Contributor Author

@Scorpiokat Scorpiokat Oct 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I make it nil, I got a warning about a unability to use it in this case.

And a lot of tests (TestErrors, TestMainExample, TestTimeSeries, for instance) use DeafultGlyphStyle, so a lot of tests fail in this case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am talking about GlyphStyleFunc being nil. not GlyphStyle (which is indeed set by default to DefaultGlyphStyle.)

I believe that, if you leave the whole NewScatter unmodified with regard to what it was before, then everything should work well.

ie: NewScatter should look like:

// NewScatter returns a Scatter that uses the
// default glyph style.
func NewScatter(xys XYer) (*Scatter, error) {
	data, err := CopyXYs(xys)
	if err != nil {
		return nil, err
	}
	return &Scatter{
		XYs:        data,
		GlyphStyle: DefaultGlyphStyle,
	}, err
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooops, sorry. Did not understand that. Now it works with the original NewScatter.

for _, p := range pts.XYs {
c.DrawGlyph(pts.GlyphStyle, vg.Point{X: trX(p.X), Y: trY(p.Y)})
for i, p := range pts.XYs {
if pts.GlyphStyleFunc != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of testing over and over whether GlyphStyleFunc is nil or not, couldn't we create a little closure (before the for-loop) that will return the correct GlyphStyle value?

something like:

glyph := func(i int) draw.GlyphStyle { return pts.GlyphStyle }
if pts.GlyphStyleFunc != nil {
    glyph = pts.GlyphStyleFunc
}
for i, p := range pts.XYs {
    c.DrawGlyph(glyph(i), vg.Point{X: trX(p.X), Y: trY(p.Y)})
}

Copy link
Contributor Author

@Scorpiokat Scorpiokat Oct 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do absolutely agree with you, have made the changes.

@Scorpiokat
Copy link
Contributor Author

As for a separate color scatter plot, I am happy to bring it back, as it looks much better on its own, than together with 3 others.

@sbinet
Copy link
Member

sbinet commented Oct 27, 2017

yes, I think having a separate example (and thus a separate test) would be better.

@Scorpiokat
Copy link
Contributor Author

Looks better now?

@sbinet
Copy link
Member

sbinet commented Oct 27, 2017

yes.

@@ -16,6 +16,10 @@ type Scatter struct {
// XYs is a copy of the points for this scatter.
XYs

//GlyphStyleFunc, if not nil, specifies GlyphStyles
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please leave a space between // and the start of the comment.

@@ -16,6 +16,10 @@ type Scatter struct {
// XYs is a copy of the points for this scatter.
XYs

//GlyphStyleFunc, if not nil, specifies GlyphStyles
//for individual points
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here as well.


s.GlyphStyle = DefaultGlyphStyle

s.GlyphStyleFunc = func(int) draw.GlyphStyle {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my comment above is still relevant, I believe.

"gonum.org/v1/plot/vg/vgimg"
)

// ExampleScatterColor draws some scatter points, a line,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please rename this as: ExampleScatter_color so it looks well on godoc (and is part of the examples for Scatter.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still there :)
also, the comment doesn't match what the example actually does :)

perhaps something along these lines would do ?

// ExampleScatter_color draws some scatter points.
// Each point will be plotted with a different color depending on some external criteria.
func ExampleScatter_color() {

(it could be improved, though.)


// ExampleScatterColor draws some scatter points, a line,
// and a line with points.
func ExampleScatterColor() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please rename this as: ExampleScatter_color so it looks well on godoc (and is part of the examples for Scatter.)

}

p.Add(sc)
//p.Legend.Add("",sc)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that comment could be removed, I believe.

Copy link
Contributor

@ctessum ctessum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far!


// randomPoints returns some random x, y points
// with some interesting kind of trend.
randomPoints := func(n int) XYs {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense for this function to return XYZs instead of XYs? Then we wouldn't need to separately create z values below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case we will have to include Z to the type Scatter, won't we?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can change it to be the same randomTriples function that is in the bubbles example.


z := []float64{31, 41, 51, 61, 71, 81, 91, 101, 111, 121, 131, 141, 151, 161, 171, 181}

sc.GlyphStyleFunc = func(i int) draw.GlyphStyle {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment describing what this is doing?


p.Add(sc)

// Create a legend.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be a better way to create the legend.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for example:

pLegend, err := plot.New()
if err != nil {
    log.Panic(err)
}
l := &ColorBar{ColorMap: colors}
l.Vertical = true
pLegend.Add(l)
pLegend.HideX()
pLegend.Y.Padding = 0
pLegend.Title.Text = "Title"

Then, below:

dcPlot := draw.Crop(dc, 0,-legendWidth, 0, 0)
dcLegend := draw.Crop(dc, 300-legendWidth, 0, 0, 0)
pLegend.Draw(dcLegend)
p.Draw(dcPlot)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This solution is a bit confusing, as it adds the legend (and the values which our z[i] is taking) on the Y axis (or X axis in case of the l.Horizontal option). So,I kept the previous solution for now.

@ctessum
Copy link
Contributor

ctessum commented Oct 27, 2017

One additional outstanding issue is that since the GlyphStyle also determines the size of the glyph, that needs to be accounted for in the GlyphBoxes method. You can see how this is done in the Bubbles plotter.

One way to test that this is implemented correctly would be to add a ExampleScatter_bubbles example which re-creates the ExampleBubbles using the GlyphStyleFunc. Then, if others are comfortable with it, we can delete the whole Bubbles plotter to reduce the size of the API.

@Scorpiokat
Copy link
Contributor Author

@ctessum this is the same solution, I have just added some stats to it in the last commit to show that the system builds a file that differs in size from the golden one. However, on the systems I have tested the code, these 2 files are identical, so the test does not fail. I will remove these lines once we find out why that happens.

@ctessum
Copy link
Contributor

ctessum commented Nov 18, 2017 via email

@Scorpiokat
Copy link
Contributor Author

@ctessum yes. On this machine, and on a new freshly made one.

@kortschak
Copy link
Member

I get perfect a perfect match on my machine (test passes and sha1 matches). A suggestion I have is to render the png to base64 (we have a helper for this). Then at least we can see what the image looks like. Additionally, you can make changed in cmpimg to see in detail why it's failing the check.

@Scorpiokat
Copy link
Contributor Author

@kortschak and what sha1 are you getting on your machine? Could you please send that to me? And that would be really helpful if I could have got the image that the system is generating, otherwise I cannot test it against my result. just by making commits.

@kortschak
Copy link
Member

The sha1 I get is f0217b5efc20f13c2c43a4e5a760b23eef9d1eec.

What you can do is use encoding/base64 to output a text representation of the file to the travis log. Then you can reconvert that to a binary after scraping it from the log.

@Scorpiokat
Copy link
Contributor Author

Scorpiokat commented Nov 20, 2017

@kortschak I have done that. The base64 representations are different. However, when reconverted to binary - they look the same, while sha1sum says otherwise. Looks like cmpimg needs to be inspected too.

@kortschak
Copy link
Member

kortschak commented Nov 20, 2017

I have compared the two images and they are nearly the same, but there is a very slight difference in the colour spectrum at the bottom of the plot (not visible with a straight subtraction of images, but very clear when the intensity difference is scaled by ~25x).

scattercolor_fail

scattercolor_diff

What we can see from this is that the failure is entirely from the colourbar (this is easy to check by removing that code and seeing if the failure persists). What I suspect is the underlying cause is floating point differences between the arch we are running and the arch travis is using. We've seen things like this before in Gonum proper (lapack being a good example). I don't know why it's affecting the bar and not the scatter dots, but I suspect it is the bar image scaling. If this is the problem, the obvious solution is to remove the colorbar.

@Scorpiokat
Copy link
Contributor Author

@kortschak thanks a lot for checking the images! I haven't noticed that difference in colour. Changing ColorBar back to PaletteThambnailers definitely worked for the best now. Please have a look.

Copy link
Member

@kortschak kortschak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but please wait for @ctessum.

@ctessum
Copy link
Contributor

ctessum commented Nov 23, 2017

LGTM, but remember to squash all the commits into one, or perhaps two commits: one for everything except for the last commit here, and then a second one where the colorbar is changed to the palette thumbnails.

@kortschak
Copy link
Member

It's also worth rebasing onto master (there are conflicts that will make this difficult though) since default ticks behaviour has changed.

I'd rebase -i into the structure that you want, then attempt the rebase onto master and fix any conflicts that arise.

@kortschak
Copy link
Member

There's still a lot going on in these commits that is not relevant to the actual change (left over from the debugging). To avoid a lot of messing around, I think we should just merge this into one commit. However, it still needs to be rebased onto master; the failure you see is due to a disagreement between the old and the new default ticks.

So, can you try to rebase onto the gonum plot master and then regen the golden image and push it back here.

@Scorpiokat
Copy link
Contributor Author

Dan @kortschak I have regenerated files, but the one that got regenerated is just one which now is conflicting. ScatterColor_golden has passed the tests on my machine, i.e. was not regenerated. I guess, the Travis check will be failing again once the existing conflict is resolved.

@kortschak
Copy link
Member

Travis is correct here, the current gonum/plot master is not in the history of this PR. To get this, you need to rebase onto gonum/plot's master; your master needs to be made to match the gonum/plot master.

Unfortunately you have squashed the two commits together prior to thie merge of your master (I must have been unclear - we squash when we merge). I've gone through the reflog to find that.

git checkout master
git fetch [email protected]:gonum/plot.git master
git reset --hard FETCH_HEAD
git checkout newfeature
git reset --hard 0e1f7c64cf5940fd8f59ef0254e3913c9ee66b12 # Your previous state.
git rebase master

check everything is OK

cd plotter
go test -regen
git add testdata/scatterColor_golden.png
git commit -m "fix scatterColor_golden.png"
git push -f origin newfeature

@Scorpiokat
Copy link
Contributor Author

Scorpiokat commented Nov 24, 2017

Thank you for such a detailed reply. However, not everything is still OK:

>git fetch [email protected]:gonum/plot.git master
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

@kortschak
Copy link
Member

Oh, sorry, that requires, auth use git fetch https://github.com/gonum/plot.git master instead.

@Scorpiokat
Copy link
Contributor Author

Looks like it has finally passed the checks.

@Scorpiokat
Copy link
Contributor Author

We decided to leave the ColorBar issue for later, right? Or...?

@kortschak
Copy link
Member

You mean the float errors?

@kortschak
Copy link
Member

We have 25 commits back, sorry. I am just going to leave this and squash them in the merge.

@Scorpiokat
Copy link
Contributor Author

Yes, I meant that.

@kortschak kortschak merged commit 444dee9 into gonum:master Nov 24, 2017
@kortschak
Copy link
Member

Without knowing what is going on on their hardware, there is not much we can do.

@Scorpiokat
Copy link
Contributor Author

Sure.
Thank you. I can delete the newfeature branch, I guess.

@kortschak
Copy link
Member

I should have checked. That was not rebased onto the gonum/plot master and was passing because the old default ticks algorithm was still being used. I have sent a PR fixing the breakage.

@Scorpiokat
Copy link
Contributor Author

Sorry and thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants