Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parser of html links #87

Merged
merged 22 commits into from
Feb 5, 2023
Merged

Add parser of html links #87

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
c55daf8
Add floki to dependencies
nappex Mar 27, 2022
5c36f40
Add test for parsing html links
nappex Mar 27, 2022
9c9a668
Add parser of html links
nappex Mar 27, 2022
17612b2
Merge branch 'master' into links-parser
Glutexo Jul 20, 2022
6d7a44f
Add header to toolbox dependencies
nappex Jan 21, 2023
7d55762
Move parser.ex to spider_html.ex
nappex Jan 21, 2023
fff0bb9
Rename document to hmtl
nappex Jan 21, 2023
def56d9
Make from html sigil a constant
nappex Jan 21, 2023
6422b03
Fix sigil content, indentation and remove double qoutes
nappex Jan 21, 2023
9d48de1
Add a tag without href to test if error will invoke
nappex Jan 21, 2023
14a3f1a
Fix path to module and function find_links
nappex Jan 21, 2023
f38c140
Rename var result to links
nappex Jan 21, 2023
2b8df13
Add describe to test
nappex Jan 21, 2023
d5b753c
Rename test file to spider_html, to keep changes from parser to spider
nappex Jan 21, 2023
5068ba6
Merge branch 'master' into links-parser
nappex Jan 21, 2023
af9da82
Fix slash from left to right
nappex Jan 21, 2023
57f681c
Update mix.exs suitable name of toolbox
nappex Feb 5, 2023
ae8dc30
Update test/spider_html_test.exs
nappex Feb 5, 2023
4a53137
Update test/spider_html_test.exs
nappex Feb 5, 2023
485c904
Fix floki definition for mix.exs to be consistent
nappex Feb 5, 2023
792a512
Update test/spider_html_test.exs
nappex Feb 5, 2023
36a9324
Add other tag with href than 'a' to test if we collect href only for …
nappex Feb 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions lib/parser.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
defmodule Parser do
def html_links(document) do
Floki.parse_document!(document)
|> Floki.find("a")
|> Floki.attribute("href")

This comment was marked as resolved.

Sign in to view

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤨

nappex marked this conversation as resolved.
Show resolved Hide resolved
end
end
5 changes: 4 additions & 1 deletion mix.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ defmodule Onigumo.MixProject do
# {:dep_from_hexpm, "~> 0.3.0"},
# {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"},
{:httpoison, "~> 1.8"},
{:mox, "~> 1.0", only: :test}
{:mox, "~> 1.0", only: :test},

# Toolbox dependencies
nappex marked this conversation as resolved.
Show resolved Hide resolved
{:floki, "~> 0.32.0"}
Glutexo marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Owner

@Glutexo Glutexo Jan 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{:floki, "~> 0.32.0"}
{:floki, "~> 0.32.1"}

Do we want to specify the latest version at the creation of the pull request or round it down to zero? What is the convention?

Answer: ~> means >= the specified patch version, but == the minor version. I would use 0.32.1 now, but we can bump all versions in a separate pull request.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, you mentioned that in #161 (comment).

]
end

Expand Down
2 changes: 2 additions & 0 deletions mix.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
%{
"certifi": {:hex, :certifi, "2.8.0", "d4fb0a6bb20b7c9c3643e22507e42f356ac090a1dcea9ab99e27e0376d695eba", [:rebar3], [], "hexpm", "6ac7efc1c6f8600b08d625292d4bbf584e14847ce1b6b5c44d983d273e1097ea"},
"floki": {:hex, :floki, "0.32.1", "dfe3b8db3b793939c264e6f785bca01753d17318d144bd44b407fb3493acaa87", [:mix], [{:html_entities, "~> 0.5.0", [hex: :html_entities, repo: "hexpm", optional: false]}], "hexpm", "d4b91c713e4a784a3f7b1e3cc016eefc619f6b1c3898464222867cafd3c681a3"},
Glutexo marked this conversation as resolved.
Show resolved Hide resolved
"hackney": {:hex, :hackney, "1.18.0", "c4443d960bb9fba6d01161d01cd81173089686717d9490e5d3606644c48d121f", [:rebar3], [{:certifi, "~>2.8.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~>6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~>1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~>1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.3.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~>1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~>0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "9afcda620704d720db8c6a3123e9848d09c87586dc1c10479c42627b905b5c5e"},
"html_entities": {:hex, :html_entities, "0.5.2", "9e47e70598da7de2a9ff6af8758399251db6dbb7eebe2b013f2bbd2515895c3c", [:mix], [], "hexpm", "c53ba390403485615623b9531e97696f076ed415e8d8058b1dbaa28181f4fdcc"},
"httpoison": {:hex, :httpoison, "1.8.0", "6b85dea15820b7804ef607ff78406ab449dd78bed923a49c7160e1886e987a3d", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "28089eaa98cf90c66265b6b5ad87c59a3729bea2e74e9d08f9b51eb9729b3c3a"},
"idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~>0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"},
"metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], [], "hexpm", "69b09adddc4f74a40716ae54d140f93beb0fb8978d8636eaded0c31b6f099f16"},
Expand Down
25 changes: 25 additions & 0 deletions test/parse_test.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
defmodule ParseTest do
use ExUnit.Case

@urls [
nappex marked this conversation as resolved.
Show resolved Hide resolved
"http://onigumo.local/hello.html",
"http://onigumo.local/bye.html"
]

test("Parsing values of href attributes in html links") do
document = ~s("<!doctype html>
nappex marked this conversation as resolved.
Show resolved Hide resolved
<html>
<body>
<section id="content">
<p class="headline">Floki</p>
<a href="http://onigumo.local/hello.html">Hello</a>
<a href="http://onigumo.local/bye.html">Bye</a>
<span data-model="user">onigumo</span>
</section>
</body>
</html>")
nappex marked this conversation as resolved.
Show resolved Hide resolved
nappex marked this conversation as resolved.
Show resolved Hide resolved

result = Parser.html_links(document)
nappex marked this conversation as resolved.
Show resolved Hide resolved
assert result == @urls
end
end