Welcome to moddb’s documentation!¶
The moddb python library is a module to scrape the ModDB.com website. The goal is to emulate the moddb website using function and OOP as if ModDB was simply an API. Ideally, in its finished state you should be able to pair with a web framework to perfectly emulate the website and all its features.
This library is not endorsed by the ModDB team and remains a scraper. As such please be respectful of their servers when using it.
Basic Usage¶
The simplest way to use this library is to simply pass a ModDB url to the parse function and let the magic happen:
import moddb
mod = moddb.parse_page("http://www.moddb.com/mods/edain-mod")
print(mod.name) #Edain Mod
Advanced Usage¶
Of course, relying on the library isn’t always what we want, you’d be forced to use requests plus nothing’s perfect and errors might occur. In some cases, you might benefit from having a finer control upon the process, in which case, it is good to know that you can easily do this with few additional imports. Given you suddenly want to use something like aiohttp to get the pages you could easily do something like this:
import moddb
import aiohttp
async def get_page():
async with aiohttp.ClientSession() as session:
async with session.get("http://www.moddb.com/mods/edain-mod") as response:
soup = moddb.soup(await r.text())
return moddb.Mod(soup)
This would return a mod object, with you only having to import the bit you want to change from the original library, in that case aiohttp. This is not fully supported for all things however, for example trying to get the articles for that mod page you would be forced to completly parse the result page on your own or be forced to use requests.
See more snippets there: Examples and Snippets.
Searching¶
The package supports searching moddb with an optional query, filters and to sort those results. Filters differ widely depending on which model is being searched therefore a good read of the documentation of the search function is recommended. The only required parameter of the search function is a valid moddb.SearchCategory as the first argument. All other arguments are optional. the search function returns a moddb.Search object. The most basic example is:
from moddb import search, SearchCategory
search = search(SearchCategory.mods)
#returns a search objects with the results being a
#list of mod thumbnails
You can also search titles of the models with a string query:
from moddb import search, SearchCategory
search = search(SearchCategory.mods, query="age of the ring")
#returns a search objects with the results being a list of
#mod thumbnails which moddb has matched with the words age of the ring
Many additional filters are available and listed in the documentation, each filter kwarg will expected a specific enum. The only exception is the game filters which expects either a game object or a moddb.Object with an id attribute. For example if we want to search for a released fantasy mod:
from moddb import search, SearchCategory, Theme, Status
search = search(SearchCategory.mods,
theme=Theme.fantasy, released=Status.released
)
#returns a search objects with the results being a list of mod
#thumbnails of mods which have been released and are labelled as
#fantasy themed.
And finally, you can sort the results with the sort kwargs which expects a tuple with the first element being a sort category which is valid for the model you’re sorting for and the second being whether you want the sort to be ascending or descending:
from moddb import search, SearchCategory, Sort
search = search(SearchCategory.mods, sort=("rating", "asc"))
#returns a search objects with the results being a list of mod
#thumbnails sorted by rating with highest rating first
Logging¶
This package makes use of the powerful Python logging module. Whenever the scrapper is unable to find a specific field for a model, either because it does not exist or because the scrapper cannot see it with its current permissions, then it will log it as an info. All logs will be sent to the moddb logger, to have access to the stream of logs you can do things like:
import logging
logging.basicConfig(level=logging.INFO)
or:
import logging
logger = logging.getLogger('moddb')
logger.setLevel(logging.INFO)
handler = logging.FileHandler(filename='moddb.log', encoding='utf-8', mode='w')
handler.setFormatter(logging.Formatter('%(asctime)s:%(levelname)s:%(name)s: %(message)s'))
logger.addHandler(handler)
Account¶
This package allows users to login into ModDB in order to have access to additional information which Guest users (the account the package uses by default) cannot see such as Guest comments waiting for approval or private groups/members. In order to login simply use the built-in library method:
import moddb
moddb.login("valid_username", "valid_password")
The session the package uses will be updated and all further requests will now be made logged in as that user. The function will raise a ValueError if the login fails. You can also log out of the account to disable access to those restricted elements with the logout function:
import moddb
moddb.logout()
Installing¶
Install with pip:
pip install moddb