A different dulwich error using Fedora 23 #75

ghost · 2016-04-03T18:34:58Z

I've generated a 64-bit Fedora 23 workstation virtual machine to try (among other reasons) to get closer to what other ofCourse developers may be using in their own builds.

I end up with this traceback:

Traceback (most recent call last):
  File "/home/joe/hfoss-S16/ofcourse_environment/bin/ofcourse", line 10, in 
    sys.exit(cli())
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/__init__.py", line 204, in openshift
    app_url = push(appname, api, domain)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 119, in push
    map(os.remove, openshift_files.keys())
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 79, in __exit__
    six.reraise(exc_type, value, tb)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 101, in push
    repo.do_commit("Commit openshift files")
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/repo.py", line 565, in do_commit
    committer = self._get_user_identity()
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/repo.py", line 496, in _get_user_identity
    config.get(("user", ), "name"),
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/config.py", line 380, in get
    raise KeyError(name)
KeyError: 'name'

To better document and make consistent what I'm doing, I made the attached script to run through the whole process of generating a test course, and captured a run of it in the second attached file using 'script'.

script.txt
output.txt

ryansb · 2016-04-19T20:43:37Z

It seems like your gitconfig lacks a user/name. Try git config --global user.name "Dee Joe"

ryansb · 2016-08-18T12:55:48Z

This is due to a missing git username. Reopen if that doesn't fix it.

ghost · 2016-08-18T13:54:04Z

I'm getting this stack trace

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 159, in openshift
    if is_dirty():
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 130, in is_dirty
    s = git.status(repo)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 549, in status
    tracked_changes = get_tree_changes(r)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 564, in get_tree_changes
    index = r.open_index()
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/repo.py", line 725, in open_index
    return Index(self.index_path())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/index.py", line 203, in __init__
    self.read()
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/index.py", line 223, in read
    f = SHA1Reader(f)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/pack.py", line 1360, in __init__
    self.sha1 = sha1('')
TypeError: Unicode-objects must be encoded before hashing

when running any one of several variants of this script:

GHACCT=ritjoe
COURSE_NAME=testofcourse
SEMESTER=F16
COURSEDIR=${COURSE_NAME}-${SEMESTER}
GHURL=https://github.com/$GHACCT/$COURSE_NAME
#USERNAME="D Joe"
#EMAIL="[email protected]"

mkdir $COURSEDIR
cd $COURSEDIR

virtualenv --no-site-packages -p python3 ofcourse_environment
source ofcourse_environment/bin/activate
pip install ofcourse
mkdir $COURSE_NAME

cd $COURSE_NAME
ofcourse new
git init
git config --replace-all --global user.name "Joe"
git config --replace-all --global user.email "[email protected]"
git add .
git commit -m "initial commit"
ofcourse openshift --app $COURSE_NAME --domain ritjoe

It's working up through the openshift deployment.

As per #75 (comment) I've played around with how I quote user.name and user.email but it seems not to matter.

ghost · 2016-08-18T13:56:15Z

I'm not sure I can re-open this myself:

http://stackoverflow.com/a/21333938

ryansb · 2016-08-18T13:57:37Z

That appears to be a dulwich bug - it's not resulting from things ofcourse passes in, but rather choking on decoding the sha (inside plumbing).

ghost · 2016-08-18T14:11:39Z

pip freeze output, as requested via IRC

$ pip freeze
ansible==2.1.1.0
Babel==2.3.4
ecdsa==0.13
httplib2==0.9.2
iniparse==0.4
Jinja2==2.8
MarkupSafe==0.23
paramiko==1.16.0
pyasn1==0.1.9
pycrypto==2.6.1
pygpgme==0.3
pyliblzma==0.5.3
python-keyczar==0.71rc0
pytz==2016.6.1
PyYAML==3.11
rpm-python==4.13.0rc1
six==1.10.0
virtualenv==14.0.6
ESC[33mYou are using pip version 8.0.2, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.ESC[0m

ryansb · 2016-08-18T14:12:24Z

dulwich isn't in there - is it living outside your virtualenv or something?

ghost · 2016-08-18T14:22:55Z

Not sure when the script bombs out of the venv, but making sure I'm in the venv and trying the ofcourse openshift step again interactively I get what looks like the same trace, but with this pip freeze:

ESCkjoe@localhost:~/scripted-ofcourse/testofcourse-F16/testofcourseESC\(ofcourse_environment) [joe@localhost testofcourse]$ pip freeze
click==6.6
dulwich==0.10.0
feedparser==5.2.1
Flask==0.11.1
Flask-Mako==0.4
Frozen-Flask==0.12
icalendar==3.10
itsdangerous==0.24
Jinja2==2.8
Mako==1.0.4
MarkupSafe==0.23
ofcourse==0.2.5
oshift==0.2.3
python-dateutil==2.5.3
pytz==2016.6.1
PyYAML==3.11
requests==2.11.1
six==1.10.0
tornado==4.4.1
validator.py==1.2.5
Werkzeug==0.11.10

ghost · 2016-08-18T14:36:33Z

Confirmed that it was just that the subshell from my script that was in the venv and that therefor left the venv as the subshell exited on finishing the script.

So, that pip freeze is representative of what is in the venv

ghost · 2016-08-19T14:11:27Z

issuing

pip install --upgrade dulwich

from within the venv takes dulwich from 0.10.0 to 0.14.1 which moves us past the error above, but then generates

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 159, in openshift
    if is_dirty():
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 131, in is_dirty
    return any(s.staged.values() + [s.unstaged])
TypeError: unsupported operand type(s) for +: 'dict_values' and 'list'

That can then be ameliorated by making the line in question:

    return any([s.staged.values()] + [s.unstaged])

Allows things to move a bit further, before running into this error:

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 204, in openshift
    app_url = push(appname, api, domain)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 93, in push
    git.reset(repo, "hard")
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 558, in reset
    tree = r[committish].tree
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/repo.py", line 449, in __getitem__
    type(name).__name__)
TypeError: 'name' must be bytestring, not str

and that's as far as I've gotten.

It looks like they've been working on this in dulwich, see this latest commit: Fix python3 compatibility

But I think the most recent release (from early July) does not yet include the python3 fix

ghost · 2016-08-20T16:15:27Z

More from our IRC conversation, with regard to forcing an even more recent version of Dulwich with python3 fixes (see above), pending its release:

You can use the git URL in your requirements file
to get the dulwich devel version with Py3 fixes
locally you can do 
pip install -U --force  'git+https://github.com/jelmer/dulwich.git#egg=dulwich'

ryansb · 2016-08-22T12:39:42Z

Great, that looks like the best solution for now. Once a new release is out, I'll pin the requirements.txt to the newer version for Python 3 installations.

ghost · 2017-08-25T13:33:14Z

OK, this saw me through last year.

That said, looks like we should put this to bed one way or another now. With the new year starting and some changes in my own environment, I'm taking a look at this again.

On the one hand, so far as I know, no one is using ofcourse this semester so we have some leisure to spend time looking at this instead of OMG PLEASE JUST MAKE IT WORK.

On the other hand, pressures off and easy to forget in light of other priorities.

ghost mentioned this issue Aug 18, 2016

Version skew with pypi, where did this print statement come from? #76

Closed

ryansb closed this as completed Aug 18, 2016

ryansb reopened this Aug 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A different dulwich error using Fedora 23 #75

A different dulwich error using Fedora 23 #75

ghost commented Apr 3, 2016

ryansb commented Apr 19, 2016

ryansb commented Aug 18, 2016

ghost commented Aug 18, 2016

ghost commented Aug 18, 2016

ryansb commented Aug 18, 2016

ghost commented Aug 18, 2016

ryansb commented Aug 18, 2016

ghost commented Aug 18, 2016

ghost commented Aug 18, 2016

ghost commented Aug 19, 2016

ghost commented Aug 20, 2016

ryansb commented Aug 22, 2016

ghost commented Aug 25, 2017

A different dulwich error using Fedora 23 #75

A different dulwich error using Fedora 23 #75

Comments

ghost commented Apr 3, 2016

ryansb commented Apr 19, 2016

ryansb commented Aug 18, 2016

ghost commented Aug 18, 2016

ghost commented Aug 18, 2016

ryansb commented Aug 18, 2016

ghost commented Aug 18, 2016

ryansb commented Aug 18, 2016

ghost commented Aug 18, 2016

ghost commented Aug 18, 2016

ghost commented Aug 19, 2016

ghost commented Aug 20, 2016

ryansb commented Aug 22, 2016

ghost commented Aug 25, 2017