rdomains 0.5.0

A label has a date, and now the package says so

Two of the four lookup sources are no longer published: DMOZ closed in March 2017 and Shallalist stopped in 2022. Their labels were correct when assigned, but domains expire and change hands, so a lookup today can return the previous registrant’s category. Until now that answer was presented identically to one from a list updated last week.

• New source_vintage() reports every category source, when it was last published, whether it is still maintained, and its successor where one exists.
• shalla_cat(), dmoz_cat() and stevenblack_cat() now return a source_last_published column. For the two dead lists this is a constant; for Steven Black’s actively-maintained hosts file it is the fetched file’s own date.
• Looking up against a discontinued source warns once per session — not once per call, which is a warning people learn to filter out.

The sibling project piedomains measured what this confusion costs: its worst class disagreed with its own page content 71% of the time, and the cause was not bad annotation but roughly a decade between the label and the page. Only 60% of the domains it trained on still resolve.

Fetch page content, not just look domains up

New collect_content() fetches homepage HTML and text, so a domain can be classified on what it says today rather than on what a list said years ago. It returns one row per requested domain, never dropped, each carrying status, stage, error_code and retryable – so a transient failure is distinguishable from a permanent one and only the right rows get retried. fetch_error_codes() documents the closed set of reasons; fetch_report() summarises a run.

Supporting functions, all usable on their own if you already hold HTML:

• page_signals() reports whether a page is an anti-bot interstitial, a domain-parking placeholder, a server’s “nothing here” page, or too thin to classify. Vendor presence alone is not a block: reddit, walmart and quora all serve real pages while embedding reCAPTCHA.
• html_text_content() extracts text, title, description and language.

The crawler identifies itself as rdomains/<version> with a contact URL, obeys robots.txt including Crawl-delay, spaces requests to the same host, caps the response body, follows redirects by hand so every hop is re-validated, and refuses hosts resolving to private or link-local addresses.

Static HTML only – no headless browser, so a JavaScript-rendered page comes back thin and says so.

Bug fixes

• get_dmoz_data() built its output path with paste0(), so the documented default outdir = "." produced a hidden .dmoz_domain_category.csv that dmoz_cat() would then fail to find. It now uses file.path(), matching its two siblings.
• stevenblack_cat(use_file = NULL) re-downloaded roughly 4 MB on every call. The file is now cached for the session.

Testing

• The Shallalist and DMOZ tests download real files; they now carry skip_on_cran() and skip_if_offline() guards, as CRAN policy requires.

rdomains 0.4.0

Breaking Changes

• Removed get_alexa_data() function (Alexa service discontinued by Amazon)

Major Changes

• Removed unused aws.alexa dependency
• Removed devtools from Imports (incorrect usage)
• Added modern tidyverse-style API with comprehensive input validation
• Significant code deduplication through shared helper functions

API Updates

• Updated virustotal_cat() to use VirusTotal API v3 (previously v2.0)
• Updated documentation references to v3 API endpoints
• Fixed virustotal_cat() implementation to properly extract categories from v3 API response structure

Improvements

• All categorization functions now validate inputs with helpful error messages using cli package
• Standardized parameter naming (virustotal_cat now uses ‘domains’ instead of ‘domain’)
• Better error messages with clear guidance on how to fix issues
• Modernized code style (pipes, purrr, tibble internally with data.frame output for compatibility)
• Improved file path handling with informative errors
• Enhanced rate limiting in LLM functions
• Cleaner domain preprocessing logic shared across all functions

Internal Changes

• Added helper functions for common operations:
  • clean_domains() - standardized domain cleaning
  • validate_domains() - comprehensive input validation
  • validate_data_file() - consistent file validation
  • get_api_key() - unified API key retrieval
  • build_categorization_prompt() - LLM prompt construction
  • apply_rate_limit() - rate limiting logic
• Refactored to use purrr instead of for-loops where appropriate
• All functions now return tibbles for modern data handling
• Added checkmate for robust input validation
• Added readr for faster CSV reading
• Extracted domain cleaning logic to single function
• Improved string operations with stringr
• Removed redundant :: notation for imported functions (cleaner code, consistent with @importFrom)

Breaking Changes

• All categorization functions now return tibbles instead of data.frames
• get_alexa_data() has been removed (service discontinued)
• Input validation is now stricter (NULL and empty strings are properly rejected)
• virustotal_cat() parameter renamed from domain to domains for consistency

rdomains 0.3.0

• NEW: Added LLM-based domain classification with openai_cat() and claude_cat() functions
• Support for OpenAI GPT models and Anthropic Claude models for domain categorization
• Flexible custom category schemas - users can specify their own categories or use defaults
• Consistent API design matching existing *_cat() functions for seamless integration
• Built-in rate limiting and error handling for API calls
• REMOVED: BrightCloud support due to service unavailability
• Updated documentation URLs from HTTP to HTTPS where applicable
• Fixed Shallalist references to reflect service discontinuation

rdomains 0.2.1

• shallalist stopped its service so downloaded latest shalla db and changed the URL from which we fetch the shallalist file

rdomains 0.2.0

• URL fixes. in resubmission now because site from which data was downloaded went down which broke some tests

rdomains 0.1.9

• R package supporting headless browsing has been abandoned. So removing trusted_cat. Sigh.

rdomains 0.1.8

• Function for checking if domain a university domain using https://github.com/Hipo/university-domains-list

rdomains 0.1.7

• Changes due to move to a new repo.
• Basic brightcloud function added

rdomains 0.1.6

• Adds not_news classifier that classifies not news based on published work.
• passes expect_lint_free

rdomains 0.1.5

• Shallalist and DMOZ data read in with stringAsFactors as FALSE.
• Swapped the DMOZ data to domain level category data, included English translations of non-English categories, quote protection of multiple categories.
• Accounting for changes in RSelenium — startServer() for instance is deprecated. But currently only allow for passing of log for trusted_cat.
• Fixed bug in shalla_cat for multiple domain names arguments
• Fixed small issue with adult_ml1_cat() whose returned data.frame had a column that was a named list. The column is now a vector.
• If an unknown domain is passed to virustotal, it will return an empty data.frame rather than throw an error.

rdomains 0.1.0

• Initial release