Making things, writing code, wasting time...

Tag: Raspberry Pi (Page 1 of 2)

Creating a Python app for Destiny – Part 7: Authenticating our app with Bungie.net OAuth

Introduction:

Bungie are moving away from their Cookie based authentication flow and have created a new OAuth 2.0 style flow, this allows for a safer, more standardised approach to authenticating users with Bungie.net. Read the Bungie.net authentication release article here.

Bungie outline some of the benefits of using the new authentication flow:

  • It uses OAuth 2.0 style sign in flow, and does not depend on fragile cookies.
  • It gives the user a chance to review the scope of permissions granted to an application so they can understand what the application can do on their behalf.
  • Users can review all write operations performed by an application, and disable apps they no longer wish to have access.
  • It’s sanctioned by Bungie, and we will smile upon apps that use this mechanism instead of cookies. Also, this is the only mechanism permitted by our terms of service for application developers to make use of APIs that require authentication.

If you’d like to review the previous Destiny API posts, check them out here:

  1. Send a request to read Xurs inventory.
  2. Send a HTML formatted email with Xurs inventory.
  3. Log into Destiny and Bungie.net via your PSN account.
  4. Transfer items from your vault to your character and equip them.
  5. Reading a characters inventory and vault contents.
  6. Creating a Python web server with Flask.

You can also find me on Twitter here @Allyn_H_

Also, many thanks to all the people involved in the Destiny Item Manager development team (too many to name individually) and Vlad from Destiny Trials Report,  for their help debugging some of the issues with the OAuth flow!

Creating an app on Bungie.net:

To create an app on Bungie, you need to visit the developer page here. If you’ve previously created one, you’ll need to update it for the new authentication flow.

Fill in the required details:

  1. Application name: Give your app a recognisable name.
  2. Application status: Set it to private unless you plan to deploy this app.
  3. Website: A place where people can find details of your app.
  4. Redirect URL: You’ll need to set a HTTPS redirect URL for your app in order to complete OAuth authorisation.
Setting Callback URL and permissions.

I’ve added all the permissions here, as I’m also using this to test other features – you may not need all these permissions.

Then click on the save changes button at the bottom.

Hooray, your app is created!

App is created, now let’s get coding!

Now, let’s get to coding up the flow…

Creating the authorisation URL:

When you registered your app on Bungie, you should have been given an authorisation URL with a unique number, like this: https://www.bungie.net/en/Application/Authorize/1234.

We are going to take that URL and add some parameters to it before we make our authorisation request. Here’s what the updated “index.html” route looks like:

AUTH_URL = 'https://www.bungie.net/en/Application/Authorize/1234?'

@app.route('/')
@app.route('/index')
def index():
    state = make_authorization_url()
    state_params = {'state': state}
    url = AUTH_URL + urllib.urlencode(state_params)
    return render_template('index.html', url=url)

Before we look at what the state parameter is doing, here’s how we pass the URL to our HTML “index.html” template, via the render_template method.

 Creating the index page:

I’m going to build on the templates made in the previous blog post, you can review them here. The index.html page is left deliberately bare, there’s a lot of cool stuff we could do here but let’s get our basic program up and running first. In our index.html template file, we can create a hyperlink to authorise our account like this:

{% block page_content %}

<div>
    <h2><a href="{{ url }} " >Authenticate with Bungie</a></h2></br></br>
</div>


{% endblock %}

The value for {{ url }} will be passed in from the render_template method, and the hyper link will be created.

Preventing Cross Site Request Forgery with the state parameter:

As an extra precaution, we are going to add some Cross Site Request Forgery (CSRF) protection, as an added security benefit to the visitors of our site, there is a great post about this, referenced on the Bungie authentication article, you can read the post here.

Here’s how we are going to handle the state parameter of our authorisation request and callback:

  1. Create a state value, a random string of numbers and letters.
  2. Save this state value in the users secure HTTPS session.
  3. Add this parameter to the authorisation URL when we direct the user to the Bungie authorisation URL.
  4. When the user is redirected to back to our site via the callback URL, Bungie will echo the state parameter back to us.
  5. Compare the state parameter from Bungie and the state parameter we stored in the users session.

If the state parameters are the same – we know the user is who they say they are, and nothing dodgy is happening.

Here’s what the state parameter looks like in action:

State parameter in action during app authorisation.

Here we can see the state parameter in the URL.

Here’s what it looks like when Bungie redirects the user to the callback URL:

State parameter in callback.

User is redirected to the callback URL, then redirected to the /index route.

Let’s code this up –  here’s what our index view looks like (deliberately basic):

@app.route('/')
@app.route('/index')
def index():
    state = make_authorization_url()
    state_params = {'state': state}
    url = AUTH_URL + urllib.urlencode(state_params)
    print url
    return render_template('index.html', url=url)

The first thing we do is call the function make_authorization_url() and store the result in the “state” variable.

Inside the make_authorization_url(), we create a unique UUID (Universally Unique Identifier)  32 bit string, using  the Python library “uuid4”.  We store this UUID string in a variable called “state”.

Now that we have generated the state string, we’ll need to store, so we then call a function, save_created_state(state) and pass it the “state” value we just generated. We also return the

Here’s what the code looks like:

 
def make_authorization_url():
    # Generate a random string for the state parameter
    from uuid import uuid4
    state = str(uuid4())
    save_created_state(state)
    return state 

Now that we have generated the “state” value, we need to store this in the session, here’s what the code for save_created_state() looks like:

# Save state parameter used in CSRF protection: 
def save_created_state(state):
    session['state_token'] = state
    pass

Now that we have generated the state parameter, we can generate our authorisation URL and send the user to Bungie to be authenticated.

Handling a callback:

Once we have sent our request for authorisation, our server will listen on the callback URL for a response. The response from Bungie will also echo our CSRF state parameter back to us, so we are sure the response is from Bungie.

The callback route should read the state parameter, test to see if it’s the same parameter we stored in the users session – if it’s not the same state parameter- we send a HTTP 403 “forbidden” response. If the state parameter is the same as the one we sent, we can then store the authorisation code (access_token).

@app.route('/callback/bungie')
def bungie_callback():
    state = session.get('state_token')
    if not is_valid_state(state):
        print "Uh-oh, this request wasn't started by us!"
        abort(403)
    session.pop('state_token', None)
    code = request.args.get('code')
    access_code = code
    token = get_token(code)
    return redirect(url_for('index'))

The method  is_valid_state(state) is used to check that the state echoed back from Bungie is the same as the state value we have stored in our session.

Below you can see we are passing in the returned state value and comparing it to the “saved_state” value taken from the session:

def is_valid_state(state):
    saved_state = session['state_token']
    if state == saved_state:
        print "States match, you are who you say you are!"
    return True
    else:
        return False

If the states match, we return “True” and send our request for the access_token, if the states do not match, we return “False” and throw our 403 error.

The code “session.pop(‘state_token’, None)” removes the state parameter from the session (set’s it to “None”) – we don’t need it anymore.

Getting the Access Token from the Authorisation Code:

Now that the user has authorised their account and we have received the authorisation code from Bungie.net, we can swap this authoirsation code for the access token. This relates to step 4 & 5 of the Bungie.net authorisation flow.

First, we take the authorisation code and add it as the value of a Python dictionary object “post_data” – this dictionary object will be added to the body of the post request and transmitted as a JSON object.

def get_token(code):
    HEADERS = {"X-API-Key":'MY-API'}
    post_data = {'code': code}
    response = requests.post(access_token_url, json=post_data, headers=HEADERS)
    :

Now that we have made a request for the token – lets look at the response:

We've got the access_token!

JSON response containing access_token and refresh_token.

The section we are most interested in (right now) is:

    "accessToken": {
        "readyin": 0, 
        "expires": 3600, 
        "value": "COoJEo ... j8w=="
 }

We now have the access token needed to make any authorised API request! The response contains a bit more information too; the access token is ready in 0 seconds – which means we can use it immediately. The access token will expire in 3600 seconds – 1 hour from now. The access token value, is of course, the really long string.

Now that we have the JSON response, we can save the access token and the refresh token. I’m also saving the time values which tell me when my refresh token is ready and when it expires.

I’m not going to worry about refreshing the token in this blog post, as it takes a bit more work. I will be updating my code and writing a new blog post at a later date.

def get_token(code):
    :
    access_token = response.json()['Response']['accessToken']['value']
    refresh_token = response.json()['Response']['refreshToken']['value']
    refresh_ready = datetime.now() + timedelta(seconds=int(response.json()['Response']['refreshToken']['readyin']))
    refresh_expired = datetime.now() + timedelta(seconds=int(response.json()['Response']['refreshToken']['expires']))
    save_session(token_json)
    userSummary = GetCurrentBungieUser(oauth_session)
    return userSummary.json()['Response']['displayName']

We will use this to create an authorised HTTP session.

Creating our authorised session:

Now that we have our API key and our access token – we can create an authorised session. As before the API-Key is added to the session header “X-API-Key”. We also need to add an “Authorization” header with the value of our access token, here’s how the access token above would look: “Bearer  COoJEo … j8w==” (I’ve shortened it a little here).

Here’s the code to create the session:

def save_session(token_json):
    oauth_session = requests.Session()
    oauth_session.headers["X-API-Key"] = API_KEY
    oauth_session.headers["Authorization"] = 'Bearer ' + str(token_json)
    access_token = "Bearer " + str(token_json)

Making an authorised request:

Now that we are authorised – we can try a simple GET request to the GetCurrentBungieAccount endpoint (as recommended in the Bungie Auth article), this will give us some information on the logged in character, from their Bungie account.

For this – I’ve created a new view template called “vault.html”, for now this is just a straight copy of the “index.html” template. In the future, I’ll populate this with all of the vault contents.

def vault():
    userSummary = GetCurrentBungieAccount(oauth_session)
    return render_template('vault.html', 
        character=userSummary.json()['Response']['user']['displayName'], 
        lightLevel = charSummary.json()['Response']['data']['characterBase']['stats']['STAT_LIGHT']['value'],
        emblemImage = account.json()['Response']['data']['characters'][0]['emblemPath'],
        backgroundImage = account.json()['Response']['data']['characters'][0]['backgroundPath'],
    )

Here’s what the code for GetCurrentBungieAccount looks like, as we’ve seen before, it’s the same Python Requests GET format, we are passing the “oauth_session”, that we saved a moment ago, into the function as a parameter. We are returning the JSON response to the vault() route.

def GetCurrentBungieAccount(session):
    req_string = 'https://www.bungie.net/Platform/User/GetBungieNetUser/'
    res = session.get(req_string)
    return res

I’ve also created a new function to return some character specific data using the getCharacterSummary endpoint, this will give me some data like my characters light level and my characters emblem.

def GetCharacterSummary(session):
    req_string = base_url + membershipType + "/Account/" + destinyMembershipId + "/Character/" + characterId + "/"
    res = session.get(req_string)
    return res

Viewing our web site:

As usual, the full set of code can be found on my GitHub page:

https://github.com/AllynH/Destiny_Flask_Webserver

There are a few files / folders, so you’ll need to download it from there.

To run the code you can type:

python app.py

This will start the Flask Web Server, you’ll see some output like this:

* Restarting with stat
Opening Manifest...
Finished!
 * Debugger is active!
* Running on https://127.0.0.1:5000/ (Press CTRL+C to quit)

First, click on the “Authenticate with Bungie” link and follow the instructions to authenticate your account.

Index page view:

Here is the view of the index page.

You should see a screen like this, asking you to review and approve the permissions required by this app.

Authorise your app.

Don’t forget to review what permissions your giving the app.

You should then be redirected back to the index page.

Now that you’ve been authorised and logged into Bungie via our app, we can open the link “view your vault contents”, you’ll see something like this:

Welcome to your vault - yet to be finished.

Gamertag, light level, Emblem and emblem background display.

Next steps:

Right now, we’ve created the basic shell of the website / app. This is a good start and we can build onto this, add features, create new pages.

The next steps for me will be to add more functionality to the website:

  • Character inventory view.
  • Xur inventory view.
  • Finish the refresh token flow, so users don’t need to authenticate every 30 mins.
  • Add a user database to securely store the users refresh tokens.
  • Add a transfer item feature.

All of the above is finished (except the transfer item feature), I just need to write up the blog post 🙂 I’m looking to deploy the code soon to a real live web site, so stay tuned.

 

Creating a Python app for Destiny – Part 6: Creating a Python web server with Flask

Introduction:

In my previous blog posts, I’ve been working to build a wep app based on Bungies API for Destiny the game. I’ve made great progress with the server side requests and processing the data responses from Bungie, the next steps are to integrate this code into a web server. This will also work out well as I move my code over to the new Bungie OAuth fow.

After a bit of research, I’ve decided to go with the Flask microframework. As their tag line suggests, Flask is a very light weight microframework and appears to be a little easier than Django to setup for smaller projects. In order to learn more about Flask, I’ve been working from Miguel Grinbergs book and his massive online tutorial, both of which I would highly recommend. This blog post is designed as a general overview of Flask, Jinja2, Python as there are many other better thought out learning material on these subjects I’d like to get stuck into the OAuth and adding Destiny features into the code.

If you’d like to review the previous Destiny API posts, check them out here:

  1. Send a request to read Xurs inventory.
  2. Send a HTML formatted email with Xurs inventory.
  3. Log into Destiny and Bungie.net via your PSN account.
  4. Transfer items from your vault to your character and equip them.
  5. Reading a characters inventory and vault contents.

Hello World:

To get started, lets start with the “Hello World!” example on the Flask homepage.

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
 app.run()

What the code is doing:

from flask import Flask
app = Flask(__name__)

We are importing the Flask Class from the flask package. We then create an instance of a Flask object and name it “app”. We pass “__name__” as the first parameter to our Flask object, this is the “import_name” parameter and you can read more about it here.

@app.route("/") 
def hello(): 
    return "Hello World!" 

The @app.route() decorator is used to tell Flask what URL will trigger our function. In this case, a visitor to our homepage will trigger the function hello(). Within the hello() function, we are returning the string “Hello World!”.

if __name__ == "__main__":
    app.run()

When you execute the Python code from the command line, “__name__” will be set to “__main__”, this will execute the code to run our instance of the Flask object, called “app”.

When we execute the code it will print something like this to the screen:

> python app.py
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [25/Jan/2017 12:45:43] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2017 12:45:43] "GET /favicon.ico HTTP/1.1" 404 -

When we navigate to address in our browser we will see this:

Flask Hello World app

This is what our app looks like in our web browser.

If we view the source of the web page (Right mouse button -> View source), we can see. There is no HTML displayed, just the string “Hello World!”:

Flask Hello World app - page source

No HTML here – just a string.

This is a good start, we have created a web server with 7 lines of code.

Adding features to our server:

Now that we have a very basic web server up and running, lets add some features.

Flask_script Manager:

Flask Manager provides support for writing external scripts in Flask, in our case it will allow us to add parameters for the server directly from the command line. This will allow us to switch between debug mode and production mode, or to change the port and host IP numbers directly from the command line.

Here’s how we create a Manager instance and link it to our “app” object:

from flask_script import Manager
manager = Manager(app)

We can then restart our web server like so:

> python hello.py runserver

Or we could start on a different port like this:

> python hello.py  runserver -p 5001

Which would return:

 * Running on http://127.0.0.1:5001/ (Press CTRL+C to quit)

Flask-Bootstrap:

If you’re not familiar with HTML or web development, Bootstrap is a free web framework for designing websites. Bootstrap contains HTML and CSS templates used to design web pages. It really takes a lot of the pain out of designing websites 🙂

For more info on Python-Bootstrap, see here.

Here’s how we import Bootstrap into our app:

from flask_bootstrap import Bootstrap

bootstrap = Bootstrap(app)

This will allow us to create Bootstrap based designs by adding the following to our base.html template file (I’ll talk about this more in a minute):

{% extends "bootstrap/base.html" %}

HTML with the Jinja2 template engine:

In my previous posts, I’ve used a series of the file.write() commands to take the JSON response, format it as HTML , with a bit of hacking, and write it to a HTML file. This is OK for small scale programs, like the one where I send Xurs inventory as a HTML formatted email. It’s not really suitable for large scale projects, or any sort of project where you’ll be displaying multiple pages of content.

A template file is a text file that contains the response from our server, we can control the flow and contents of the template file by using Jinja2’s inbuilt Control Statements. First, let’s change the above code from returning a string – to returning a rendered Jinja2 template.

from flask import Flask, render_template

@app.route("/") def hello():
    message = "Hello, World!"
    return render_template('index.html', message=message) 

In the above code, instead of just returning the string “Hello, World!”, we are passing the Python string to the Jinja2 engine, which then renders the index.html template using the data our Python sends.

Before we build the index.html template, we can strip out all of the generic HTML code, that is reused for each page, and keep that in a base template file. This base.html template will contain our HTML <head></head> data, our CSS links, and our navigation bar.

Here’s what our base.html will look like:

When Flask-Bootstrap is initialised by our app, the template “bootstrap/base.html” becomes available to us. This “bootstrap/base.html” template contains the general HTML structure of our page and allows us to derive all of our new templates from this.

(For convenience, I’ve stripped out the navigation bar HTML, I’m using the Bootstrap example code for this).

{% extends "bootstrap/base.html" %}

{% block title %}Hello World app{% endblock %}

{% block head %}
{{ super() }}
    <link rel="shortcut icon" href="{{ url_for('static', filename='favicon.ico') }}" type="image/x-icon">
    <link rel="icon" href="{{ url_for('static', filename = 'favicon.ico') }}" type="image/x-icon">{% endblock %}
{% block navbar %}

<div class="navbar navbar-inverse" role="navigation">
    :
    :
    :
</div>

{% endblock %}

{% block content %}

<div class="container">
    {% block page_content %}{% endblock %}
</div>

{% endblock %}

Blocks like {% title%}, {% navbar %} or {% content %} are called “convenience” blocks, and are built directly into the Flask-Bootstrap, this means we can amend these blocks, without having to directly replace them. In the case of the {% head %} block, we call Flasks super() function, which allows to to amend data to the block, without overwriting the Bootstrap CSS links in the <head></head> tags.

The block we are most interested in is the {% block content %}, inside here we will find the {% block page_content %}, this will allow any template that calls our base.html to replace the contents of this block – through Template Inheritance.

{% block content %}

<div class="container">
    {% block page_content %}{% endblock %}
</div>

{% endblock %}

Here’s what our index.html will look like:

Now that all of our generic HTML has been defined in the base.html, we can create our index.html, as an extension of the base.html template.

{% extends "base.html" %}
{% block title %}Hello World app{% endblock %}

{% block page_content %}

<div>
<h1>{{ message }}</h1>
</div>

{% endblock %}

The index.html defines the contents of the page_content block, in our case we will print the string stored in the Python variable “message”, inside a set of <div> and <h1> tags:

<h1>{{ message }}</h1>
 

Folder structure:

In this example, we’ll have a base.html file for all of our generic HTML and an index.html, which will be populated by our Python code. This is a basic example, but later I’ll be building several more pages and HTML templates into this project.

Flask folder structure

Here is the folder structure Flask uses for defining template and static files.

Running the web server:

Now that the code is complete we can start our web server and see what our index page looks like.

We can start our web server like so:

> python hello.py runserver
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [25/Jan/2017 12:45:43] "GET / HTTP/1.1" 200 -

Then navigating to http://127.0.0.1:5000/ we can view our active web server:

Web server with HTML output.

Web server shown on the left and HTML source code on the right.

As we can see, we are no longer outputting just a text string, we are outputting a full HTML web page, created from our base.html, index.html and the output of our Python code.

Download the code:

The full set of Python code can also be found on my GitHub page here: https://github.com/AllynH/Flask_webserver

As there are a few files in this project, it’s best to copy it from there.

Next steps:

Now that we have a web server built in Python using Flask we can:

  • Update the code with the new Bungie OAuth login flow.
  • Build in our previous Bungie code.

The new Bungie authentication flow!

Hi there,

As of the 14th of December, Bungie have changed their authentication flow for the Destiny API. Bungie are moving away from the cookie based authentication flow, to the OAuth 2.0 flow.

The code in the previous blog posts will still work, but Bungie will soon be switching over fully to the new authentication system.

This is good news for anyone using the apps based on the Destiny API, as it is much more secure, offers the users more control over how their data is used, and what can controlled from external applications.

You can read about the changes from the official Bungie release here:

The new Bungie OAuth flow.

I’m currently creating the Python code for the new authentication flow. The code is all working but I’m taking a week off to spend Christmas with my family, so stay tuned for the blog post with updated code 🙂

Creating a Python app for Destiny – Part 5: Reading a characters inventory and vault contents.

Introduction:

In the previous sections I showed how to:

  1. Send a request to read Xurs inventory.
  2. Send a HTML formatted email with Xurs inventory.
  3. Log into Destiny and Bungie.net via your PSN account.
  4. Transfer items from your vault to your character and equip them.

In this section I’m going to lay the foundations of a full inventory management system. This section will detail how to read the list of items that are currently equipped on your character and how to read the contents of your vault.

I’ll also show how, by downloading the full Destiny Manifest, we can speed up our code and drastically reduce the amount of HTTP requests we need to send to Bungie.

Also, I like to say a big thanks to all of the people on Reddit, Twitter and Bungie.net for their help and interest 🙂

You can find me on Twitter here @Allyn_H_

The getCharacterInventory endpoint:

The BungieNetPlatform wiki for this endpoint can be found here. This endpoint will return a JSON response with a list of each of the equipped items for a given character ID. It works in a similar way to the Xur adviser endpoint (seen in parts 1 & 2). The request will return a series of encoded hashes, which we’ll then need to send as a series of separate requests to query the Destiny manifest, in order to decrypt this data into a human readable format.

  • This is a public endpoint – no need to be logged in, anyone can use this.
  • This a HTTP GET request – you don’t need to send a JSON package.
  • Your destinyMembershipId needs to be attached to the URL as a parameter – more info here.
  • Your characterId needs to be attached to the URL as a parameter – more info here.

Making the request:

First we build the endpoint URL and make the GET request:

base_url = "https://www.bungie.net/platform/Destiny/"
req_string = base_url + membershipType + "/Account/" + destinyMembershipId + "/Character/" + charId + "/Inventory"
res = session.get(req_string)

The request URL will look like this:

https://www.bungie.net/platform/Destiny/2/Account/4611686018436136301/Character/2305843009222916165/Inventory/

Parsing the response:

Next we need to parse through the multidimensional JSON response and pick out the relevant data:

for equipment in res.json()['Response']['data']['buckets']['Equippable']:
    for item in equipment['items']:
        print "itemHash is: \t\t", item['itemHash']
        print "itemInstanceId is: \t", item['itemInstanceId']
        hashReqString = base_url + "Manifest/6/" + str(item['itemHash'])
        res2 = requests.get(hashReqString, headers=HEADERS)
        item_name = res2.json()['Response']['data']['inventoryItem']['itemName']
        item_tier = res2.json()['Response']['data']['inventoryItem']['tierTypeName']
        item_type = res2.json()['Response']['data']['inventoryItem']['itemTypeName']
        item_icon = res2.json()['Response']['data']['inventoryItem']['icon']
        print "Item name is: " + item_name
        print "Item type is: " + item_tier + " " + item_type
        print "Item icon is: http://www.bungie.net" + item_icon

The above code will read through the large JSON response and print out the data we want. For each item in our inventory, we will print out the items name, its type, the tier type (common, legendary or exotic) its description and its icon. The dict item [‘Equippable’] is an array, so we need to loop through each piece of “equipment” and pick out each of the [‘items’] stored there.

The code will return something like this:

itemHash is: 1703777169
itemInstanceId is: 6917529081613012276
Item name is: 1000-Yard Stare
Item type is: Legendary Sniper Rifle
Item icon is: http://www.bungie.net/common/destiny_content/icons/c1c49acd0fd146d7b32184f23c64dfe5.jpg

The GetVault endpoint:

The BungieNetPlatform wiki for this endpoint can be found here. This endpoint also works in a similar way to the Xur adviser endpoint (seen in parts 1 & 2) and the getCharacterInventory endpoint, it will return a series of encoded hashes, which we’ll then need to send as a series of separate requests to query the Destiny manifest. If our vault was full, we currently have space for 288 items. This means we’ll need to make 289 (1 request to the vault + 288 requests to decrypt items ) requests to the Bungie servers to get all the data we want! Making that many requests takes way too much time and eats way too much data, I’ll explain how to remove these requests in a bit 🙂

  • This is a private endpoint – you need to be logged in, with a persistent HTTP session.
  • This is a GET request – you don’t need to send a JSON package.
  • Your destinyMembershipId needs to be attached to the URL as a parameter – more info here.

Making the request:

The code for this request will look like this:

getVault_url = base_url + membershipType + "/MyAccount/Vault/"
res = session.get(getVault_url, params={'accountId': destinyMembershipId})

Parsing the response:

Next we need to parse through the multidimensional JSON response and pick out the relevant data:

for bucket in vaultResult.json()['Response']['data']['buckets']:
    for item in bucket['items']:
        print item['itemHash']
        print item['itemInstanceId']
        hashReqString = base_url + "Manifest/6/" + str(item['itemHash'])
        res2 = requests.get(hashReqString, headers=HEADERS)
        myItem = item['itemHash']
        item_name = res2.json()['Response']['data']['inventoryItem']['itemName']
        item_tier = res2.json()['Response']['data']['inventoryItem']['tierTypeName']
        item_type = res2.json()['Response']['data']['inventoryItem']['itemTypeName']
        item_icon = res2.json()['Response']['data']['inventoryItem']['icon']
        print "Item name is: " + item_name
        print "Item type is: " + item_tier + " " + item_type
        print "Item icon is: http://www.bungie.net" + item_icon

The above code will read through the large JSON response and print out the data we want. For each item in our inventory, we will print out the items name, its type, the tier type (common, legendary or exotic) its description and its icon. The dict item [‘buckets’] is an array, so we need to loop through each “bucket” and pick out each of the [‘items’] stored there. The [‘buckets’] item is a storage category in your vault, such as Armour, Weapons, Consumables, Shaders, Emblems, etc. Each of the [‘items’] refers to the item stored in that vault space.

It’s also important to remember, we need to collect the “itemInstanceId” for each item, as this is required to transfer the item to and from the vault.

hashReqString = base_url + "Manifest/6/" + str(item['itemHash'])
res2 = requests.get(hashReqString, headers=HEADERS)

The above lines makes each of the requests to the manifest to decrypt the “itemsHash” into a human readable format. This request is made for each of the items in our vault. This is the real time consuming part of the program.

Downloading the manifest:

So we’ve figured out how to find our vault contents and list the equipped items on our characters. We’ve also seen, for decrypting and reading the contents of our vault, we need to make 289 requests (more if Bungie ever decide to increase the vault capacity). This takes a huge chunk of time. I measured the amount of time it took to read my inventory and to send 267 requests (I had 266 items in my inventory), the script took 2 mins and 43 seconds to finish!

So in order to speed things up, I downloaded the Destiny Manifest database onto my local computer, the code and directions can be found here on the destiny devs page.

The code can be downloaded here.

Once downloaded you can execute the code like so:

python manifest_destiny.py

The output looks something like this:

Output of the script used to download the Destiny Manifest.

Output of the script used to download the Destiny Manifest.

This will download the full Manifest and store it in a Pickle file. This is also my first time using Pickle files, so I assume this is done as it’s easier to get the data from a Pickle file than an SQL database.

Here’s what they look like:

Copies of the Destiny Manifest and the Manifest as a Pickle file.

Copies of the Destiny Manifest and the Manifest as a Pickle file.

The files are 53MB and 71MB, so they’re big enough. The file sizes can be reduced by commenting out any of the Manifest items you don’t need – for example I downloaded second copy of the Manifest with only the “itemHash” data included.

Here’s how I edited the code:

Changing the code to download only the itemHash data.

Changing the code to download only the itemHash data.

Here are the file sizes:

Copies of the itemHash manifest and Pickle file.

Copies of the itemHash manifest and Pickle file.

We can see here, the Pickle file we need has reduced from 71MB to 37MB. This would also further reduce the run time of our programs. As memory isn’t really an issue for me I will be sticking to the full Manifest.

Reading the vault contents using the manifest:

Now that we have saved the Manifest as a Pickle file, we can pull the required data directly from this file instead of needing to make hundreds of HTTP requests every time we want to read our inventory or our vault contents.

To import the data from our Pickle file we use the following code:

import pickle

with open('manifest.pickle', 'rb') as data:
    all_data = pickle.load(data)

Now the object all_data will contain the entire Manifest contents.

Previously we made requests to the Manifest via a HTTP request:

hashReqString = base_url + "Manifest/6/" + str(item['itemHash'])
res2 = requests.get(hashReqString, headers=HEADERS)
myItem = item['itemHash']
item_name = res2.json()['Response']['data']['inventoryItem']['itemName']

Where we pass the parameter “6”, which refers to the definition type “InventoryItem” and the specific “itemHash”.
Now we can read the item information from the nested dict object stored in all_data:

inventoryItem = all_data['DestinyInventoryItemDefinition'][item['itemHash']]
item_name = inventoryItem['itemName']

Updating the code to pull from our local Manifest copy:

Here’s how I modified the code used to parse our vault to pull the data directly from our manifest.pickle file instead of making a HTTP request:

for bucket in vaultResult.json()['Response']['data']['buckets']:
    for item in bucket['items']:
        weapon_list[array_size]['itemReferenceHash'] = item['itemHash']
        weapon_list[array_size]['itemId'] = item['itemInstanceId']
        inventoryItem = all_data['DestinyInventoryItemDefinition'][item['itemHash']]
        item_name = inventoryItem['itemName']
        item_tier = inventoryItem['tierTypeName']
        item_type = inventoryItem['itemTypeName']
        item_icon = inventoryItem['icon']
        print "Item name is: " + item_name
        print "Item type is: " + item_tier + " " + item_type
        print "Item icon is: http://www.bungie.net" + item_icon

Run time improvements:

We can see from the table below that downloading the Manifest and removing the need for ~290 additional HTTP requests can drop the run time from 220 seconds down to 16 seconds. A lot of 16 seconds seems to be from parsing the Pickle file, which again could be optimised by either using the raw SQL data or storing it in some other format. For now though the convenience of using the Pickle file far out weighs the downsides.

Manifest Type:

Run time:

Multiple HTTP requests to Bungie Manifest 220.0 seconds
manifest.pickle – full download 22.3 seconds
item.pickle – only item 16.2 seconds

Viewing the vault contents as a HTML file:

As a quick and easy way of outputting the vault contents into a HTML file, I used the same template file and code format as used in the guide for emailing Xurs inventory.

Here are the steps:

  • Append the HTML header to the top of the my_html string.
  • Modify the code used to parse the vault contents to output a HTML formatted string.
  • Append the HTML footer to the bottom of the my_html string.
  • Concatenate the header, vault contents and footer HTML strings into the my_html string and output this to a file called vault_contents.html.

Parsing the response as HTML:

for bucket in vaultResult.json()['Response']['data']['buckets']:
    for item in bucket['items']:
        inventoryItem = all_data['DestinyInventoryItemDefinition'][item['itemHash']]
        item_name = inventoryItem['itemName']
        item_tier = inventoryItem['tierTypeName']
        item_type = inventoryItem['itemTypeName']
        item_icon = "http://www.bungie.net/" + inventoryItem['icon']
        print "Item name is: " + item_name
        array_size += 1
        print "Item is: " + item_name
        print "Item type is: " + item_tier + " " + item_type + "\n"
        my_html = my_html + "\t\t<div class=\"col-md-4\">\n"
        my_html = my_html + "\t\t\t<div class=\"thumbnail\">\n"
        my_html = my_html + "\t\t\t\t<a href=\"" + item_icon + "\">\n"
        my_html = my_html + "\t\t\t\t<img src=\"" + item_icon + "\">\n"
        my_html = my_html + "\t\t\t\t</a>\n"
        my_html = my_html + "\t\t\t\t<h3>" + item_name + "</h3>\n"
        my_html = my_html + "\t\t\t\t<p>" + item_tier + " " + item_type + "</p>\n"
        my_html = my_html + "\t\t\t</div>\n"
        my_html = my_html + "\t\t</div>\n"

Here’s what the output of the code looks like:

Even have the VoG boots. #Y1RaidStruggles

Vault contents as formatted HTML.

Running the code:

The full set of Python code can also be found on my GitHub page here: https://github.com/AllynH/Destiny_Read_Characters_Inventory_Vault

As there are a few files you’ll need to copy it from there,. In order to make the code work for you – you’ll need to input your username, password, api_key, destinyMembershipId and characterId in the Header_file.py.

Here are the values you’ll need to change:

# PSN Username:
username = emailaddr
password = mypassword

# Destiny API X-Key:
API_KEY = ""

# Destiny parameters:
membershipType = "2" # PS4 = 2
destinyMembershipId = ""

characterId = ""

You can then run the code like so:

> python Read_Inventory_and_Vault.py

As always, I’ll try to keep the GitHub repo up to date with any changes I make.

Next steps:

The code as it is, does what we want, reads the contents of our vault and prints it to a HTML file, but it’s not really a fully functioning inventory management system.

To create a fully working web application the code will need to be built into a web framework, luckily for me I’ve been reading up on the Python Flask web framework for the last few weeks 🙂

Creating a Python app for Destiny – Part 4: Transferring and equipping items.

Introduction:

In the previous sections I showed how to:

  1. Send a request to read Xurs inventory.
  2. Send a HTML formatted email with Xurs inventory.
  3. Log into Destiny and Bungie.net via your PSN account.

I’m now going to build on the previously created code to create an app that can transfer an item to and from the vault, once the item has been transferred – we can then equip it.

If you’ve been reading along with me up to this point, you should understand all of this, if not – now is a good time to check out my previous posts.

Here’s the code working (sorry about the camera shaking, it was made using my phone ):

Destiny the game, Equipping an Item with the Destiny API. from Allyn H on Vimeo.

Also, I like to say a big thanks to all of the people on Reddit, Twitter and Bungie.net for their help and interest 🙂

You can find me on Twitter here @Allyn_H_

The transferItem endpoint:

The Destiny API uses web server endpoints to execute commands, the endpoint used to transfer items is: https://www.bungie.net/platform/Destiny/transferItem/

The BungieNetPlatform wiki for this endpoint can be found here. We can see this method is very different from the Xur advisors endpoint we used in the previous examples.

Here are the main differences:

  • We still need to use the X-API key in our header.
  • We also have to attach our x-csrf token in our header.
  • This is a private endpoint – we need to be logged in to use this method.
    • We need to attach the required cookies to our POST request.
  • This is a POST method – we will need to send data with this request.
    • We need to send the data as a JSON packet.
  • The response will only tell us if we have been successful or not – the Xur advisors response was >1000 lines of text.

The code for storing the transferItem endpoint variable will look like this:

base_url = "https://www.bungie.net/platform/Destiny/"
req_string = base_url + "TransferItem/"

Important note: If the trailing “/” is missing from the “transferItem/” endpoint – the request will fail. I’ve spent way longer than I’m willing to admit debugging that issue :/

Creating the POST request payload:

The “transferItem” endpoint takes a JSON payload as an input. This JSON payload contains a lot of information. So lets take a look at how it’s put together.

The data gathered to populate this payload was found by looking at items I had equipped using the “GetCharacterInventory” endpoint, you can find more info here. This will list the relevant information for all of your equipped items, let me know if you want to see the code for this.

Name

Description

membershipType

This will be 2 for PSN and 1 for Xbox live.

itemReferenceHash

This is the “itemHash” number from our previous examples. This refers to the generic item for example: 1000-Yard Stare.

itemId

This is a unique item number relating to a specific item, that you own. For example: Allyn’s 385 light level 1000-Yard Stare with the Ambush Scope, Quickdraw and Firefly.

stackSize

How many items to transfer. Should be “1” for equipable items.

characterId

The characterId the item is being moved to or from. You can find the characterId here.

transferToVault

Move the item to or from the vault; true or false

Here’s how this method expects the JSON data to be formatted:

This is the BungieNetPlatform example for the transferItem POST payload

This is the BungieNetPlatform example for the transferItem POST payload

The good news here is it’s pretty easy to put this together in Python by creating a Dictionary object.

Making a POST request:

We’ve already looked at making HTTP GET requests with Python, but the Python Requests library also makes it really easy to make a POST request.

You can view the Requests documentation here. Here’s the quick example of how to send some data formatted as a Python dictionary:

Python Requests POST example

Python Requests POST example

Or the Requests library will also automatically encode a Python dict into a JSON object for you, when you use the “json” parameter:

Python Requests POST JSON example

Python Requests POST JSON example

Now that we know how to create a Python POST request and what data the transferItem endpoint are looking for, lets turn that into some code!

Moving an item from your vault to your inventory:

As mentioned above – the data gathered to populate this payload was found by looking at items I had equipped by using the “GetCharacterInventory” endpoint, you can find more info here. I’m planning on putting together a blog post on this too but let me know if you’d like to see the code for this.

First we create our Python dictionary object:

text_payload = {
    "membershipType": 2,
    "itemReferenceHash": 1519376148, # The Ram
    "itemId": 6917529085991104887, # The Ram
    "characterId": characterId_Warlock,
    "stackSize": 1,
    "transferToVault": False
}

Next, we convert the dictionary object into a JSON object:

payload = json.dumps(text_payload)

Now we can make the actual POST request to the Destiny servers to transfer our item.

base_url = "https://www.bungie.net/platform/Destiny/"
req_string = base_url + "TransferItem/"
res = session.post(req_string, data=payload)

The JSON response should look like this:

{
    "ThrottleSeconds": 0, 
    "ErrorCode": 1, 
    "ErrorStatus": "Success", 
    "Message": "Ok", 
    "Response": 0, 
    "MessageData": {}
}

If the item was not found in your vault, you’ll get a response like this:

{
    "ThrottleSeconds": 0, 
    "ErrorCode": 1623, 
    "ErrorStatus": "DestinyItemNotFound", 
    "Message": "The item requested was not found.", 
    "Response": 0, 
    "MessageData": {}
}

For any other issues, for example if you were to omit the trailing “/” from “https://www.bungie.net/platform/Destiny/TransferItem/” URL, the request would fail but you wouldn’t receive a JSON response, instead your code would error and you’d receive a HTTP 405 status code. You can find the status code by using the following code:

print res.status_code
405

If you see something like that coming up, you’ll need to review and fix your code.

Equipping an item from your inventory:

Now that the item is in our characters inventory – we can equip the item on that character.

The endpoint used to equip an item from your inventory is: https://www.bungie.net/platform/Destiny/EquipItem/ you can find more information about this endpoint here.

Just a note: this can only be done when in a social space or in orbit (otherwise it’d be really easy to cheat in PvP – can you imagine if you could do this in Trials???). Otherwise the action will fail and return an error message.

Creating the POST request payload:

The POST request to equip an item is smaller and we already have the details of the item we want to equip.

text_equip_payload = {
    "membershipType": 2,
    "itemId": 6917529085991104887, # The Ram
    "characterId": characterId_Warlock
}
equip_payload = json.dumps(text_equip_payload)

 

equip_url = base_url + "EquipItem/"
res = session.post(equip_url, data=equip_payload)

Again, if the item was not found, you’ll get a response like this:

{
    "ThrottleSeconds": 0, 
    "ErrorCode": 1623, 
    "ErrorStatus": "DestinyItemNotFound", 
    "Message": "The item requested was not found.", 
    "Response": 0, 
    "MessageData": {}
}

If you’re trying to equip an exotic and you already have one equipped, you’ll see an error message like this:

{
    "ThrottleSeconds": 0, 
    "ErrorCode": 1641, 
    "ErrorStatus": "DestinyItemUniqueEquipRestricted", 
    "Message": "You can only have one item of this type equipped.", 
    "Response": 0, 
    "MessageData": {}
}

Running the code:

Here is the full set of Python code, this can be copied into a file called “equipItem.py” and executed from the command prompt like so:

> python equipItem.py

The code can also be found on my GitHub page here: https://github.com/AllynH/Destiny_Equip_Item

As always, I’ll try to keep the GitHub repo up to date with any changes I make.

Creating a Python app for Destiny – Part 3: Logging in to Bungie.net and authenticating with PSN

Introduction:

In the previous sections I showed how to:

  1. Send a request to read Xurs inventory.
  2. Send a HTML formatted email with Xurs inventory.

I want to build on the previously created code to create an app that can transfer an item to and from the vault, and equip items.

In order to do that, our code will need to log in to Bungie.net and authenticate the account with PSN.

Logging in to Bungie.net and authenticating with PSN:

We are going to use the Python “Requests” package to login to Bungie.net by using our PSN account details and OAuth 2.0 to authenticate our connection with PlayStation Network.,

The good people at BungieNetPlatform have put together some guides on how to connect with Bungie.net, get authenticated with PSN (or Xbox Live – but I’m on PS4) and grab the required cookies. For this example, I used the code provided by Quantum Ascend here.

You can also see his step by step instructions here.

Here are the steps to this section:

  1. Sign in on Bungie.net via PSN – this will redirect you to the PSN sign in page.
  2. Grab our PSN session ID.
  3. Login to PSN (via OAuth) using our PSN username, password and adding our session ID as a cookie.
  4. Receive PSN a unique sign in URL and updated JSESSIONID.
  5. Request PSN  X-NP-GRANT-CODE, using updated JSESSIONID.
  6. Sign in to Bungie.net by adding the grant code to our original URL.
  7. Grab our Bungie.net authentication cookies.

Here is a flow chart detailing these steps:

Future War Cult colours - representing!

Bungie.net Sign-in flow chart

Request 1 is done in this way, to accommodate both Playstation and Xbox accounts to log in – however as I only have a Playstation 4, I’m not working on the Xbox live sign in, you can find code for that in the BungieNetPlatform guide here.

request1 = requests.get(BUNGIE_SIGNIN_URI, allow_redirects=True)
jsessionid0 = request1.history[1].cookies["JSESSIONID"]
params = urlparse(request1.url).query
params64 = b64encode(params)

Request 2 sends a POST request to the PSN sign in page. Our log in credentials are passed in a dictionary format, these are then form-encoded (by the Requests package as a HTML form) when the request is made. We also create a cookie with the JESSIONID we received from Request 1.

The response from Request 2, returns an updated JSESSIONID, also stored in a cookie – we save this updated value. The if statement checks for an authentication error being returned – this confirms our log in credentials were correct and no errors were returned.

request2 = requests.post(PSN_OAUTH_URI, data={"j_username": username, "j_password": password, "params": params64}, cookies={"JSESSIONID": jsessionid0}, allow_redirects=False)
if "authentication_error" in request2.headers["location"]:
    logger.warning("Invalid credentials")
jsessionid1 = request2.cookies["JSESSIONID"]

Request 3 sends a GET request to the returned PSN OAtuh sign in URL, adding the updated JSESSION ID, to the header. This will give us our x-np-grant-code.

request3 = requests.get(request2.headers["location"], allow_redirects=False, cookies={"JSESSIONID": jsessionid1})
grant_code = request3.headers["x-np-grant-code"]

The PSN OAtuh sign in URL will look something like this:

https://auth.api.sonyentertainmentnetwork.com/2.0/oauth/authorize?response_type=code&client_id=78xxx&redirect_uri=https%3a%2f%2fwww.bungie.net%2fen%2fUser%2fSignIn%2fPsnid&scope=psn:s2s&request_locale=en

Request 4 makes the final request to the Bungie.net sign in page, attaching the x-np-grant-code to the URL. The “params” function in the requests library attaches this code to the URL.

request4 = requests.get(BUNGIE_SIGNIN_URI, params={"code": grant_code})

The Bungie.net sign in URL with the x-np-grant-code attached should look something like this:

https://www.bungie.net/en/User/SignIn/Psnid?code=Nxxxh

Now that we have authorised our Bungie.net account with PSN, we can create a persistent session and send multiple requests.

Creating a persistent HTTP Session:

A persistent HTTP session is used to keep our HTTP connection alive allowing us to make multiple requests without the need to sign in and authenticate each time. This means we will only need to authorise our account once and can make multiple requests – so long as we attach the relevant authorisation data. This authorisation data is stored in the cookies and header data we send in our requests. The python requests package has a “Session” object, used for just this thing!

To create a HTTP session, we need to do 2 things:

  1. Send the required HTTP header data:
    • X-API-Key – the Application Programming Interface key we got from registering at Bungie.net.
    • x-csrf – our Cross Site Request Forgery protection token, received from Bungie.net after we have authenticated out app with PSN.
  2. Attach the required cookies with the correct, authenticated data:
    • bungled – received from Bungie.net after we have authenticated out app with PSN (This is also our x-csrf token).
    • bungleatk – received from Bungie.net after we have authenticated out app with PSN.
    • bungledid – received from Bungie.net after we have authenticated out app with PSN.

Here’s what that looks like when translated into Python code – first we create a requests Session object:

session = requests.Session()

Next, we add our X-API-KEY and x-csrf token to the session header:

session.headers["X-API-Key"] = API_KEY
session.headers["x-csrf"] = request4.cookies["bungled"]

Then we create our Cookies and attach them to the requests session object:

session.cookies.update(
 {
    "bungleatk": request4.cookies["bungleatk"], 
    "bungled": request4.cookies["bungled"], 
    "bungledid": request4.cookies["bungledid"]
 })

That’s it! We’re done – our app can now log into Destiny via PSN. This will allow us to use all of the private endpoints provided by the API and do lots of cool stuff, such as transferring items, equipping items, locking items, etc.
I’ll build on this code again in my next blog post.

Running the code:

Here is the full set of Python code, this can be copied into a file called “PSN_login.py”, in the same directory as your own code, and implemented like so:

from PSN_login import login

username = emailaddr
password = mypassword
api_key = API_KEY

# Log in via PSN and create our persistant HTTP session: 
session = requests.Session()
session = login(username, password, api_key)

Here’s the link to the code on my GitHub account:

https://github.com/AllynH/Destiny_Equip_Item/blob/master/PSN_login.py

Here’s the GitHub Gist:

Creating a Python app for Destiny – Part 2: Emailing Xurs inventory.

Introduction:

In the previous tutorial, I showed how you could use Python to send a request using the Destiny API, to the Bungie servers and how to decrypt the JSON reply. If you haven’t read the previous tutorial, it’s right here. I’m going to gloss over how to write the HTML, as there are plenty of really good online resources for creating and styling websites, checkout codecademy if you’re interested in a free guide.

In this tutorial I will continue on with the program we have created, to:

  1. Change our Get_Xur_inventory.py program so it writes the output as a HTML file.
  2. Have the program send this HTML, via Gmail, directly to our  email.

By writing our output as HTML we have a lot more control over the design and formatting of the email. Using HTML will also allow us to directly embed the item pictures into out email, from the URL links provided in the JSON response from Xur.

The hard part of the code is already done, with a few minor tweaks, we can have our program output Xurs inventory into a HTML formatted email.

 

Outputting Xurs inventory as HTML:

In order to create our output HTML, we are going to use a template HTML file to store as much of the generic HTML as possible, and change all of the print statements in our code to output HTML formatted data.

HTML file structure is split into 3 main parts, such as HTML version information, <head>, and <body> tags. We are going to take advantage of that and save the HTML version information and <head> section in a template HTML file, which we can reuse every time we run the script. This would allow us to update any of the generic HTML code separately – for example if you wanted to change some styling or CSS information, without messing with your Python code

We can then use our script to generate only the HTML needed to display the items from Xurs inventory, this will change every week so needs to be generated every time we run our script.

Opening our template HTML file:

The following code will open a file called “template.html” for reading, and save all of the contents into a string object called “my_html”, then close the file.

template_file = open('template.html', "r")
my_html = template_file.read() template_file.close()

Outputting HTML from our Python code:

Below is an example of how we’d like to format the HTML code, essentially we wrap it in a couple of <div>’s and display the image and text.

<div class="col-md-4">
 <div class="thumbnail">
 <a href="item_url">
 <img src="item_url">
 </a>
 <h3>Item name: item_name</h3>
 <p>Item type is: item_type<p>
 <p>Description: item_description<p>
 </div>
</div>

Here’s what the corresponding Python code looks like:

my_html = my_html + "<div class=\"col-md-4\">\n"
my_html = my_html + "\t<div class=\"thumbnail\">\n"
my_html = my_html + "\t\t<a href=\"" + item_url + "\">\n"
my_html = my_html + "\t\t<img src=\"" + item_url + "\">\n"
my_html = my_html + "\t\t</a>\n"
my_html = my_html + "\t\t<h3>" + item_name + "</h3>\n"
my_html = my_html + "\t\t<p>" + item_type + "<p>\n"
my_html = my_html + "\t\t<p>" + item_description + "<p>\n"
my_html = my_html + "\t</div>\n"
my_html = my_html + "</div>\n"

We’ve wrapped each of the lines of HTML code in a write statement. We’ve already populated all of the item_url, item_name, item_type and item_description variables with our code.

So here’s what each line is doing. In this example we are creating a <h3> heading, with the text from our “item_name”. “\t” creates a tab and “\n” moves to a new line. All of this text is concatenated into one string, which we add on to the end of the “my_html” string object.

my_html = my_html + "\t\t<h3>" + item_name + "</h3>\n"

Closing the HTML:

Now that we have created and populated <div>’s with each of Xurs items, we can add the code to close the HTML </body> and </html> tags.

my_html = my_html + "\t\t</div> <!-- row -->\n"
my_html = my_html + "\t</div> <!-- container -->\n"
my_html = my_html + "</div> <!-- inventory-container -->\n"
my_html = my_html + "</body>\n"
my_html = my_html + "</html>\n"

The “my_html” string will store all of the HTML code used in the body of the email we send.

Sending an email with Python:

Python comes installed with a number of really useful libraries, including the smtplib (Simple Mail Transfer Protocol library) and the MIMEMultipart and MIMEText (Multipurpose Internet Mail Extensions, allows text, images and other options) libraries.

For this section I referenced two tutorials, here Nael Shaib shows how to make a basic email program, and here Darren Massena shows how to create a HTML formatted email and attach external pictures.

Some important notes:

Google recently changed their security requirements, so to use this program, you will need to change your Google settings to allow less secure apps. I needed to do the following:

  1. Click here to allow less secure apps. Google will still reject access to your account, until you authorise it.
  2. Click here to authorise your app. This will authorise your app access to your Google account.
  3. Gmail and some other mail clients do not support CSS styling, so if you wanted to convert some existing CSS to inline HTML, you could use a tool like this. That being said, my iPad and Android phone both display mails with full CSS – so I’ve included some CSS styling in my code 🙂
Enable access for less secure apps.

Enable access for less secure apps.

Alright, that’s enough talk – lets get to coding!

Importing the required libraries:

First lets import the required libraries:

import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText

Next we’ll set up our email address parameters, enter your details as follows.

# Mail parameters:
fromaddr = "TO_ADDRESS"
toaddr = "FROM_ADDRESS"
password = "GMAIL_PASSWORD"

Next, lets create the email header information:

# Compose mail: 
msgRoot = MIMEMultipart() 
msgRoot['From'] = fromaddr 
msgRoot['To'] = toaddr 
msgRoot['Subject'] = "Xurs Inventory." 
msgRoot.preamble = "This is a multipart message in MIME format." 

The line:

msgRoot = MIMEMultipart()

creates an email MIMEMultipart object, we then set the [‘To’], [‘From’] and [‘Subject’] parameters of the email on the following 3 lines.

Creating the email body:

We’ve already populated the HTML into a string object called “my_html”.

The “my_html” string object is added to the body of the email with the following commands, the 2nd parameter  passed to the MIMEText object sets the email type as HTML:

msgText = MIMEText(my_html, 'html')
msgAlternative.attach(msgText)

The following commands create an SMTP mail object, connect to the Gmail SMTP server on port 587.

server = smtplib.SMTP('smtp.gmail.com', 587)
server.ehlo()
server.starttls()

The next line actually logs into our Gmail account, we pass our Gmail address and password as parameters to this.

server.login(fromaddr, password)

The next 2 lines actually send the email,

server.sendmail(fromaddr, toaddr, msgRoot.as_string())
server.quit()

Running the code:

Here is the full set of Python code, this can be copied into a file called “Get_Xur_inventory_email.py” and executed from the command prompt like so:

> python Get_Xur_inventory_email.py

The code can also be found on my GitHub page here: https://github.com/AllynH/Destiny_Get_Xur_inventory_email

As always, I’ll try to keep the GitHub repo up to date with any changes I make.

Creating a Python App for the Destiny API

Introduction:

I’m a huge fan of Bungie’s Destiny game, and have been playing it since launch. In Destiny there is a vendor, called Xur, who visits the game each week and sells exotic weapons and armour. Xur is only available from Friday to Sunday mornings, and appears in a different location each week – because of this he can be quite hard to track down. As with most people – I’m at work every Friday and usually busy with family time over the weekend, so I’m often left in a situation where I miss Xur or don’t have the time to log in and see what he’s selling.

This just wasn’t good enough, I couldn’t risk missing out on the next Gjallarhorn! So let’s built an app to find Xurs inventory each week.

 

API registration:

In order to build an app using the Bungie API, you need to create a Bungie.net account, and register as a developer. This only takes a few minutes, to register as a developer, follow this link: https://www.bungie.net/en/User/API

This will give you your unique X-API-Key. For every request we make to the Bungie servers, we need to send this API key in the HTTP header of the request.

 

App flow chart:

Finding Xurs inventory isn’t as straight-forward as expected. Bungie have implemented a method for reading Xurs inventory – but it returns the data in an encrypted format, as an item hash.

The reason behind this is, even items of the same type can have different perks. For example, each gun can have a different scope type, different perks, and different barrel modifications. So even 2 of the same guns can be completely different.

Because of this, we’ll need to send multiple requests to the Bungie servers and read multiple replies.

FWC representing!

Flow chart for the app.

 

Python HTTP requests:

Requests is a Python HTTP library, if you’re not familiar with it, start here. You may need to install it on your computer, If you have PIP installed on your computer requests can be installed by typing:

pip install requests

 

Sending our request to Xurs advisors page:

Our first HTTP request will be to Xurs Advisors page, here is the link https://www.bungie.net/Platform/Destiny/Advisors/Xur/

The X-API-Key is passed as a parameter with every HTTP request made to the Bungie servers:

HEADERS = {"X-API-Key":'MY-X-API-Key'}

 

Here is how we make the request:

xur_url = "https://www.bungie.net/Platform/Destiny/Advisors/Xur/"
print "\n\n\nConnecting to Bungie: " + xur_url + "\n"
print "Fetching data for: Xur's Inventory!"
res = requests.get(xur_url, headers=HEADERS)

It’s as simple as that.

So what are we doing? The line:

res = requests.get(xur_url, headers=HEADERS)

Makes a HTTP request to the URL stored in the “xur_url” string, the X-API-Key is also added to the request in the “headers” dictionary object.

Now the object “res” contains the JSON response received from the Bungie servers.

 

Parsing the JSON response:

The JSON response received from the request is pretty big, in my case it contained 1018 lines of text! So what do we do with this data? First things first, we need to know if our request was received and processed correctly. The JSON object contains a key called “ErrorStatus”, this key is used to store status of the request, so lets print the value of this key:

error_stat = res.json()['ErrorStatus']
print "Error status: " + error_stat + "\n"

Here is what the  code outputs:

Successful connection - whoop whoop!

Successful connection.

Here is an example of what the code would output if there was a successful request but Xur was not available (he’s only available from Friday to Sunday):

Office hours are Friday to Sunday morning!

Connection is successful but Xur is nowhere to be found.

 

Parsing the multidimensional JSON response:

Looking through the “res” JSON object, we can see a key called “itemHash”, located in res[‘Response’][‘data’][‘saleItemCategories’][‘saleItem’][‘item’][‘itemHash’], shown on line 23. This is the location of the encoded item details we are looking for!

Here is where things get tricky… JSON data is used to represent key-value pairs, called dictionaries in Python. However the item “saleItemCategories”, shown on line 11 – itself contains a series of key-value pairs. Also the item “saleItems”, shown on line 14, is also a dictionary.

Our JSON object contains nested dictionaries, also known as a multidimensional array.

Lots and lots of data!

Decoding the JSON response for Xurs inventory.

 

So in order to make a list of each of the “itemHash” values, we need a for loop to iterate through each of the “saleItemCategories” and another for loop to iterate through each of the “saleItems”.

for saleItem in res.json()['Response']['data']['saleItemCategories']:
	mysaleItems = saleItem['saleItems']
	for myItem in mysaleItems:
		hashID = str(myItem['item']['itemHash'])

 

Request 2 – decoding the item hash:

Now that we have a list of our itemHash’s – we need to make send a request to the Bungie Destiny Manifest page for each of the hashes.

This request takes the form of: http://www.bungie.net/Platform/Destiny/Manifest/{type}/{id}/

Where the {id} is the itemHash we just took from Xurs inventory.

A list of the hash {type}’s can be found here: http://bungienetplatform.wikia.com/wiki/DestinyDefinitionType but for this example, we know {type} will be “6”, as we are searching for an “InventoryItem”.

We can add on another request to our nested loops above:

base_url = "https://www.bungie.net/platform/Destiny/"
for saleItem in res.json()['Response']['data']['saleItemCategories']:
	mysaleItems = saleItem['saleItems']
	for myItem in mysaleItems:
		hashID = str(myItem['item']['itemHash'])
		hashReqString = base_url + "Manifest/" + hashType + "/" + hashID
		res = requests.get(hashReqString, headers=HEADERS)
		item_name = res.json()['Response']['data']['inventoryItem']['itemName']
		item_type = res.json()['Response']['data']['inventoryItem']['itemTypeName']
		item_tier = res.json()['Response']['data']['inventoryItem']['tierTypeName']
		print "Item is: " + item_name
		print "Item type is: " + item_tier + " " + item_type + "\n"

Here is what the output of our code looks like:

Here's what Xur is selling today.

Here’s what Xur is selling today.

And when we checkout Xurs inventory on the Bungie.net vendors page, here’s what we see:

Xurs inventory, taken from the Bungie.net vendors page.

Xurs inventory, taken from the Bungie.net vendors page.

Yes! Our Python app can now send requests to Bungie and print the contents of Xurs inventory. (Also, if you haven’t got The Ram – get it)

There’s lots of cool stuff sent in the JSON objects, including hyperlinks to the item images and details on the price of each item. For now, this is good enough for me.

Running the code:

Here is our full set code,  we can copy this into a file called “Get_Xur_inventory.py” and execute it from the command prompt like so:

> python Get_Xur_inventory.py

The code can also be found on my GitHub page here: https://github.com/AllynH/Destiny_Get_Xur_inventory

I’ll keep the repo up to date with any improvements or changes.

from lxml import html
import requests
import json

# Uncomment this line to print JSON output to a file:
#f = open('output.txt', 'w')

HEADERS = {"X-API-Key":'YOUR-X-API-Key'}

base_url = "https://www.bungie.net/platform/Destiny/"
xur_url = "https://www.bungie.net/Platform/Destiny/Advisors/Xur/"
hashType = "6"

# Send the request and store the result in res:
print "\n\n\nConnecting to Bungie: " + xur_url + "\n"
print "Fetching data for: Xur's Inventory!"
res = requests.get(xur_url, headers=HEADERS)

# Print the error status:
error_stat = res.json()['ErrorStatus']
print "Error status: " + error_stat + "\n"

# Uncomment this line to print JSON output to a file:
#f.write(json.dumps(res.json(), indent=4))

print "##################################################"
print "## Printing Xur's inventory:"
print "##################################################"

for saleItem in res.json()['Response']['data']['saleItemCategories']:
	mysaleItems = saleItem['saleItems']
	for myItem in mysaleItems:
		hashID = str(myItem['item']['itemHash'])
		hashReqString = base_url + "Manifest/" + hashType + "/" + hashID
		res = requests.get(hashReqString, headers=HEADERS)
		item_name = res.json()['Response']['data']['inventoryItem']['itemName']
		print "Item is: " + item_name
		item_type = res.json()['Response']['data']['inventoryItem']['itemTypeName']
		item_tier = res.json()['Response']['data']['inventoryItem']['tierTypeName']
		print "Item type is: " + item_tier + " " + item_type + "\n"
		

 

Here are a few ideas for the next steps:

  1. Add e-mail function, so the script mails me automatically when Xur lands.
  2. Create a web server, so we can access the app from a web page.
  3. Find Xurs location and add it to the output.
  4. Play some Trials of Osiris and get ready for the Rise Of Iron expansion

I’m hoping to expand on this program and add features as I go, leave me a comment and let me know if there’s anything you’d like to see, or if you fancy carrying me to the Lighthouse!

 

Connecting a Raspberry Pi to a WD MY Cloud Network Attached Hard Drive:

Recently during a house move, I dropped my good old reliable Raspberry Pi – hard drive Network Attached Storage unit.
Basically I connected my external hard drive to a Raspberry Pi and had a Network Attached Storage drive which allowed me to access my media from any device on my home network.

I then realised it was time to buy a dedicated NAS.

I shelled out for the WE My Cloud 4 TB server, which by all reports is a great piece of gear – however I wanted something more than a standalone NAS and I wanted to be able to access the WD My Cloud from my Pi.

Here’s how I connected my Pi to the My Cloud! The guide below should work for any NAS – not just the WD My Cloud.

Find the IP address of your NAS:

If you don’t know the IP address of your NAS, you can perform an “arp-scan” from your Raspberry Pi to find it, here’s how I found mine:

# sudo arp-scan --localnet --interface=wlan0

If you still can’t find the IP address, for the WD My Cloud you can find the IP address in your settings, as per the instructions here.

In my case my NAS IP address was: 192.168.192.62

Mounting the NAS to the Raspberry Pi:

The first step to accessing the NAS from your Raspberry Pi is to mount the external HDD as a file system on the Raspberry Pi, this will allow you to view the NAS, as you would any directory on the Pi. This is pretty easy actually, as the CIFS (CIFS Common Internet File Share, a protocol dictating how different OS’ share files between them, including Windows and Linux) protocol takes care of everything.

First make a directory for the share:

# mkdir wdmycloud

Next mount the drive using the IP address and the Raspberry Pi directory you want to mount to:

# sudo mount -t cifs -o guest //192.168.192.62/Public /home/pi/wdmycloud
mount command

mount command

In this example, I am mounting the “Public” folder located on my NAS to the wdmycloud folder located on my Raspberry Pi.

The command syntax is:
mount -t <Mount Type> -o <Access> <SOURCE> <DESTINATION>

After executing the mount command, you should now be able to access the NAS file system as you would any other directory!

Automatically mount the NAS on power up:

 

Edit the FSTAB to mount your NAS automatically on power up:

To make the mount permanent, we need to add the NAS file system to the Raspberry Pi’s /etc/fstab file – the File System Table.

# sudo nano /etc/fstab
Edit the FSTAB file.

Edit the FSTAB file.

Add the NAS as a file system in the FSTAB file.

//192.168.192.62/Public /home/pi/wdmycloud cifs guest     0       0
Adding the NAS to the FSTAB.

Adding the NAS to the FSTAB.

You can see from the last line in the FSTAB file above, I have added the NAS as a file system in my FSTAB file.

This will automatically mount the NAS every time you power up your Raspberry Pi!

Testing the NAS is connected automatically on power up:

First step, reboot your Pi… :

# sudo reboot
Reboot your Pi.

Reboot your Pi.

 

Next step, check your NAS directory from the Pi:

# cd wdmycloud
# ls -larth
Confirm the share is working.

Confirm the share is working.

Success! The mount works, I can now access all of my pictures, music and movies from my Raspberry Pi. Everything is safely stored on my WD My Cloud, which keeps 2 copies of all of my data – so if anything goes wrong, I’ll always have my data backed up 😉

 

Making a Twitter-Bot on your Galileo or Raspberry Pi

Twitter have developed an API (Application Programming Interface) for their website, which makes it really easy to send and receive Tweets from your Raspberry Pi or Galileo! The Twitter API takes all of the hard work out of writing a program to interface with Twitter, There are several ways to access the Twitter API, the easiest of which (in my opinion anyway 🙂 is to use the Twython package of the Python language.

Differences between Raspberry Pi and the Galileo:

It’s really easy to install Twython on the Raspberry Pi, a little harder to install on the Galileo – so for that reason, I’ll show the step-by-step instructions from the Galileo install. The only difference is the Galileo doesn’t require you to use the sudo command as you already have root permissions set. For example, when editing a file with the Galileo you would use:

# nano my_file.txt

When editing the file on a Raspberry Pi you would use:

# sudo nano my_file.txt

Changing the date on the Galileo:

In order to install some of these packages, you’ll need to update the date and time of your Galileo – this isn’t automatically done when you connect to the internet as you may expect. If you don’t update your date and time, you’ll get SSL certification errors when you try to download the Twython package.

To change the date and time – use the following command in the format year-month-day hour:minute:second:

# date --set="2015-01-20 10:00:00"

Installing Twython (and other Python packages):

In order to connect our computer to Twitter, we’ll need to download some Python packages:

  1. setuptools – this will allow you to “Easily download, build, install, upgrade, and uninstall Python packages.”
  2. pip – this is a  Python package installer.
  3. twython – this is the package which will actually interface with Twitter.

Here are the commands to install these packages – remember if you’re on a Raspberry Pi you’ll need to put “sudo” in front of each command.

# apt-get install python-setuptools

Here’s what that looks like on your computer, when asked “do you want to continue” as per the picture below – enter “y”. to continue.

Package 1 of 3.

Installing the setuptools Python package.

Next, install the “pip” package:

# easy_install pip

Finally, using pip – install the Twython package:

# pip install twython

 

Creating a Twitter App:

To create a Twitter App, you’ll need to sign up to Twitter and register the account as an application. This is important as you’ll need to verify this App with Twitter every time you use it. The verification method Twitter uses is called OAuth 2.0 to verify your App, this means you’ll never have to supply your password to 3rd party App developers but it does make it a little harder to verify your App – the good news is, Twython and other API’s handle all the OAuth pain, all you need to do is register your App and save the information. Register your app here: https://apps.twitter.com/ Click on “Create New App” and enter your information.

Let the fun begin!

Creating an App.

 

Changing App permissions:

As this is your App and you’ll want to be able to play around with it – you can chance the App permissions to allow you to read, write and access your direct messages. You can change these permissions at a later stage.

Change permissions.

Change permissions.

Authorize your account:

You need to create your access token and access token secret before you use your App. Click on the “Keys and access tokens” tab. Click on the “Create my access token” button.

Generate your secret token.

Generate your secret token.

You should now have all 4 pieces of required information:

  1. Consumer Key (API key).
  2. Consumer Secret (API Secret).
  3. Access Token
  4. Access Token Secret.

Now it’s time to write some Python!

Writing the Python:

Creating a Python file:

On your Raspberry Pi or Galileo, create a file called “Tweet.py” using the following command:

# nano Tweet.py

Now paste in the Python code:

Twitter Authorization:

In order to send  a Tweet, you’ll need to send Twitter your OAuth information. This process is handled by the Twython package. Here we are creating 4 string objects and a Twython object called “twitter”. When we create the Twython object we are passing the 4 strings to it an arguments. These are the access keys you generated in the previous section.

CONSUMER_KEY = '<YOUR CONSUMER_KEY>'
CONSUMER_SECRET = '<YOUR CONSUMER_SECRET>'
ACCESS_KEY = '<YOUR ACCESS_KEY>'
ACCESS_SECRET = '<YOUR ACCESS_SECRET>'
twitter = Twython(CONSUMER_KEY,CONSUMER_SECRET,ACCESS_KEY,ACCESS_SECRET)

Reading in a command line argument:

To make the code a bit more flexible, we can pass the text we want to Tweet into the Python script as a command line argument. This is done by using the system “argv” parameter. In our case – we only want to take the first 140 characters of this text, as this is the character limit set by Twitter for each Tweet. We do that by using the command:

sys.argv[1][:140]

Executing the code:

You can execute the code from the command line like so:

# python Tweet.py "Hello world."

Here’s what that looks like on the Galileo:

First Tweet!

Hello World!

Then check your Twitter Bot!

« Older posts

© 2024 Allyn H

Theme by Anders NorenUp ↑