Allyn H

Making things, writing code, wasting time...

Category: Linux

Creating a Python app for Destiny – Part 2: Emailing Xurs inventory.


In the previous tutorial, I showed how you could use Python to send a request using the Destiny API, to the Bungie servers and how to decrypt the JSON reply. If you haven’t read the previous tutorial, it’s right here. I’m going to gloss over how to write the HTML, as there are plenty of really good online resources for creating and styling websites, checkout codecademy if you’re interested in a free guide.

In this tutorial I will continue on with the program we have created, to:

  1. Change our program so it writes the output as a HTML file.
  2. Have the program send this HTML, via Gmail, directly to our  email.

By writing our output as HTML we have a lot more control over the design and formatting of the email. Using HTML will also allow us to directly embed the item pictures into out email, from the URL links provided in the JSON response from Xur.

The hard part of the code is already done, with a few minor tweaks, we can have our program output Xurs inventory into a HTML formatted email.


Outputting Xurs inventory as HTML:

In order to create our output HTML, we are going to use a template HTML file to store as much of the generic HTML as possible, and change all of the print statements in our code to output HTML formatted data.

HTML file structure is split into 3 main parts, such as HTML version information, <head>, and <body> tags. We are going to take advantage of that and save the HTML version information and <head> section in a template HTML file, which we can reuse every time we run the script. This would allow us to update any of the generic HTML code separately – for example if you wanted to change some styling or CSS information, without messing with your Python code

We can then use our script to generate only the HTML needed to display the items from Xurs inventory, this will change every week so needs to be generated every time we run our script.

Opening our template HTML file:

The following code will open a file called “template.html” for reading, and save all of the contents into a string object called “my_html”, then close the file.

template_file = open('template.html', "r")
my_html = template_file.close()

Outputting HTML from our Python code:

Below is an example of how we’d like to format the HTML code, essentially we wrap it in a couple of <div>’s and display the image and text.

<div class="col-md-4">
 <div class="thumbnail">
 <a href="item_url">
 <img src="item_url">
 <h3>Item name: item_name</h3>
 <p>Item type is: item_type<p>
 <p>Description: item_description<p>

Here’s what the corresponding Python code looks like:

my_html = my_html + "<div class=\"col-md-4\">\n"
my_html = my_html + "\t<div class=\"thumbnail\">\n"
my_html = my_html + "\t\t<a href=\"" + item_url + "\">\n"
my_html = my_html + "\t\t<img src=\"" + item_url + "\">\n"
my_html = my_html + "\t\t</a>\n"
my_html = my_html + "\t\t<h3>" + item_name + "</h3>\n"
my_html = my_html + "\t\t<p>" + item_type + "<p>\n"
my_html = my_html + "\t\t<p>" + item_description + "<p>\n"
my_html = my_html + "\t</div>\n"
my_html = my_html + "</div>\n"

We’ve wrapped each of the lines of HTML code in a write statement. We’ve already populated all of the item_url, item_name, item_type and item_description variables with our code.

So here’s what each line is doing. In this example we are creating a <h3> heading, with the text from our “item_name”. “\t” creates a tab and “\n” moves to a new line. All of this text is concatenated into one string, which we add on to the end of the “my_html” string object.

my_html = my_html + "\t\t<h3>" + item_name + "</h3>\n"

Closing the HTML:

Now that we have created and populated <div>’s with each of Xurs items, we can add the code to close the HTML </body> and </html> tags.

my_html = my_html + "\t\t</div> <!-- row -->\n"
my_html = my_html + "\t</div> <!-- container -->\n"
my_html = my_html + "</div> <!-- inventory-container -->\n"
my_html = my_html + "</body>\n"
my_html = my_html + "</html>\n"

The “my_html” string will store all of the HTML code used in the body of the email we send.

Sending an email with Python:

Python comes installed with a number of really useful libraries, including the smtplib (Simple Mail Transfer Protocol library) and the MIMEMultipart and MIMEText (Multipurpose Internet Mail Extensions, allows text, images and other options) libraries.

For this section I referenced two tutorials, here Nael Shaib shows how to make a basic email program, and here Darren Massena shows how to create a HTML formatted email and attach external pictures.

Some important notes:

Google recently changed their security requirements, so to use this program, you will need to change your Google settings to allow less secure apps. I needed to do the following:

  1. Click here to allow less secure apps. Google will still reject access to your account, until you authorise it.
  2. Click here to authorise your app. This will authorise your app access to your Google account.
  3. Gmail and some other mail clients do not support CSS styling, so if you wanted to convert some existing CSS to inline HTML, you could use a tool like this. That being said, my iPad and Android phone both display mails with full CSS – so I’ve included some CSS styling in my code 🙂
Enable access for less secure apps.

Enable access for less secure apps.

Alright, that’s enough talk – lets get to coding!

Importing the required libraries:

First lets import the required libraries:

import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText

Next we’ll set up our email address parameters, enter your details as follows.

# Mail parameters:
fromaddr = "TO_ADDRESS"
toaddr = "FROM_ADDRESS"
password = "GMAIL_PASSWORD"

Next, lets create the email header information:

# Compose mail: 
msgRoot = MIMEMultipart() 
msgRoot['From'] = fromaddr 
msgRoot['To'] = toaddr 
msgRoot['Subject'] = "Xurs Inventory." 
msgRoot.preamble = "This is a multipart message in MIME format." 

The line:

msgRoot = MIMEMultipart()

creates an email MIMEMultipart object, we then set the [‘To’], [‘From’] and [‘Subject’] parameters of the email on the following 3 lines.

Creating the email body:

We’ve already populated the HTML into a string object called “my_html”.

The “my_html” string object is added to the body of the email with the following commands, the 2nd parameter  passed to the MIMEText object sets the email type as HTML:

msgText = MIMEText(my_html, 'html')

The following commands create an SMTP mail object, connect to the Gmail SMTP server on port 587.

server = smtplib.SMTP('', 587)

The next line actually logs into our Gmail account, we pass our Gmail address and password as parameters to this.

server.login(fromaddr, password)

The next 2 lines actually send the email,

server.sendmail(fromaddr, toaddr, msgRoot.as_string())

Running the code:

Here is the full set of Python code, this can be copied into a file called “” and executed from the command prompt like so:

> python

The code can also be found on my GitHub page here:

As always, I’ll try to keep the GitHub repo up to date with any changes I make.

Creating a Python App for the Destiny API


I’m a huge fan of Bungie’s Destiny game, and have been playing it since launch. In Destiny there is a vendor, called Xur, who visits the game each week and sells exotic weapons and armour. Xur is only available from Friday to Sunday mornings, and appears in a different location each week – because of this he can be quite hard to track down. As with most people – I’m at work every Friday and usually busy with family time over the weekend, so I’m often left in a situation where I miss Xur or don’t have the time to log in and see what he’s selling.

This just wasn’t good enough, I couldn’t risk missing out on the next Gjallarhorn! So let’s built an app to find Xurs inventory each week.


API registration:

In order to build an app using the Bungie API, you need to create a account, and register as a developer. This only takes a few minutes, to register as a developer, follow this link:

This will give you your unique X-API-Key. For every request we make to the Bungie servers, we need to send this API key in the HTTP header of the request.


App flow chart:

Finding Xurs inventory isn’t as straight-forward as expected. Bungie have implemented a method for reading Xurs inventory – but it returns the data in an encrypted format, as an item hash.

The reason behind this is, even items of the same type can have different perks. For example, each gun can have a different scope type, different perks, and different barrel modifications. So even 2 of the same guns can be completely different.

Because of this, we’ll need to send multiple requests to the Bungie servers and read multiple replies.

FWC representing!

Flow chart for the app.


Python HTTP requests:

Requests is a Python HTTP library, if you’re not familiar with it, start here. You may need to install it on your computer, If you have PIP installed on your computer requests can be installed by typing:

pip install requests


Sending our request to Xurs advisors page:

Our first HTTP request will be to Xurs Advisors page, here is the link

The X-API-Key is passed as a parameter with every HTTP request made to the Bungie servers:

HEADERS = {"X-API-Key":'MY-X-API-Key'}


Here is how we make the request:

xur_url = ""
print "\n\n\nConnecting to Bungie: " + xur_url + "\n"
print "Fetching data for: Xur's Inventory!"
res = requests.get(xur_url, headers=HEADERS)

It’s as simple as that.

So what are we doing? The line:

res = requests.get(xur_url, headers=HEADERS)

Makes a HTTP request to the URL stored in the “xur_url” string, the X-API-Key is also added to the request in the “headers” dictionary object.

Now the object “res” contains the JSON response received from the Bungie servers.


Parsing the JSON response:

The JSON response received from the request is pretty big, in my case it contained 1018 lines of text! So what do we do with this data? First things first, we need to know if our request was received and processed correctly. The JSON object contains a key called “ErrorStatus”, this key is used to store status of the request, so lets print the value of this key:

error_stat = res.json()['ErrorStatus']
print "Error status: " + error_stat + "\n"

Here is what the  code outputs:

Successful connection - whoop whoop!

Successful connection.

Here is an example of what the code would output if there was a successful request but Xur was not available (he’s only available from Friday to Sunday):

Office hours are Friday to Sunday morning!

Connection is successful but Xur is nowhere to be found.


Parsing the multidimensional JSON response:

Looking through the “res” JSON object, we can see a key called “itemHash”, located in res[‘Response’][‘data’][‘saleItemCategories’][‘saleItem’][‘item’][‘itemHash’], shown on line 23. This is the location of the encoded item details we are looking for!

Here is where things get tricky… JSON data is used to represent key-value pairs, called dictionaries in Python. However the item “saleItemCategories”, shown on line 11 – itself contains a series of key-value pairs. Also the item “saleItems”, shown on line 14, is also a dictionary.

Our JSON object contains nested dictionaries, also known as a multidimensional array.

Lots and lots of data!

Decoding the JSON response for Xurs inventory.


So in order to make a list of each of the “itemHash” values, we need a for loop to iterate through each of the “saleItemCategories” and another for loop to iterate through each of the “saleItems”.

for saleItem in res.json()['Response']['data']['saleItemCategories']:
	mysaleItems = saleItem['saleItems']
	for myItem in mysaleItems:
		hashID = str(myItem['item']['itemHash'])


Request 2 – decoding the item hash:

Now that we have a list of our itemHash’s – we need to make send a request to the Bungie Destiny Manifest page for each of the hashes.

This request takes the form of:{type}/{id}/

Where the {id} is the itemHash we just took from Xurs inventory.

A list of the hash {type}’s can be found here: but for this example, we know {type} will be “6”, as we are searching for an “InventoryItem”.

We can add on another request to our nested loops above:

base_url = ""
for saleItem in res.json()['Response']['data']['saleItemCategories']:
	mysaleItems = saleItem['saleItems']
	for myItem in mysaleItems:
		hashID = str(myItem['item']['itemHash'])
		hashReqString = base_url + "Manifest/" + hashType + "/" + hashID
		res = requests.get(hashReqString, headers=HEADERS)
		item_name = res.json()['Response']['data']['inventoryItem']['itemName']
		item_type = res.json()['Response']['data']['inventoryItem']['itemTypeName']
		item_tier = res.json()['Response']['data']['inventoryItem']['tierTypeName']
		print "Item is: " + item_name
		print "Item type is: " + item_tier + " " + item_type + "\n"

Here is what the output of our code looks like:

Here's what Xur is selling today.

Here’s what Xur is selling today.

And when we checkout Xurs inventory on the vendors page, here’s what we see:

Xurs inventory, taken from the vendors page.

Xurs inventory, taken from the vendors page.

Yes! Our Python app can now send requests to Bungie and print the contents of Xurs inventory. (Also, if you haven’t got The Ram – get it)

There’s lots of cool stuff sent in the JSON objects, including hyperlinks to the item images and details on the price of each item. For now, this is good enough for me.

Running the code:

Here is our full set code,  we can copy this into a file called “” and execute it from the command prompt like so:

> python

The code can also be found on my GitHub page here:

I’ll keep the repo up to date with any improvements or changes.

from lxml import html
import requests
import json

# Uncomment this line to print JSON output to a file:
#f = open('output.txt', 'w')


base_url = ""
xur_url = ""
hashType = "6"

# Send the request and store the result in res:
print "\n\n\nConnecting to Bungie: " + xur_url + "\n"
print "Fetching data for: Xur's Inventory!"
res = requests.get(xur_url, headers=HEADERS)

# Print the error status:
error_stat = res.json()['ErrorStatus']
print "Error status: " + error_stat + "\n"

# Uncomment this line to print JSON output to a file:
#f.write(json.dumps(res.json(), indent=4))

print "##################################################"
print "## Printing Xur's inventory:"
print "##################################################"

for saleItem in res.json()['Response']['data']['saleItemCategories']:
	mysaleItems = saleItem['saleItems']
	for myItem in mysaleItems:
		hashID = str(myItem['item']['itemHash'])
		hashReqString = base_url + "Manifest/" + hashType + "/" + hashID
		res = requests.get(hashReqString, headers=HEADERS)
		item_name = res.json()['Response']['data']['inventoryItem']['itemName']
		print "Item is: " + item_name
		item_type = res.json()['Response']['data']['inventoryItem']['itemTypeName']
		item_tier = res.json()['Response']['data']['inventoryItem']['tierTypeName']
		print "Item type is: " + item_tier + " " + item_type + "\n"


Here are a few ideas for the next steps:

  1. Add e-mail function, so the script mails me automatically when Xur lands.
  2. Create a web server, so we can access the app from a web page.
  3. Find Xurs location and add it to the output.
  4. Play some Trials of Osiris and get ready for the Rise Of Iron expansion

I’m hoping to expand on this program and add features as I go, leave me a comment and let me know if there’s anything you’d like to see, or if you fancy carrying me to the Lighthouse!


Connecting a Raspberry Pi to a WD MY Cloud Network Attached Hard Drive:

Recently during a house move, I dropped my good old reliable Raspberry Pi – hard drive Network Attached Storage unit.
Basically I connected my external hard drive to a Raspberry Pi and had a Network Attached Storage drive which allowed me to access my media from any device on my home network.

I then realised it was time to buy a dedicated NAS.

I shelled out for the WE My Cloud 4 TB server, which by all reports is a great piece of gear – however I wanted something more than a standalone NAS and I wanted to be able to access the WD My Cloud from my Pi.

Here’s how I connected my Pi to the My Cloud! The guide below should work for any NAS – not just the WD My Cloud.

Find the IP address of your NAS:

If you don’t know the IP address of your NAS, you can perform an “arp-scan” from your Raspberry Pi to find it, here’s how I found mine:

# sudo arp-scan --localnet --interface=wlan0

If you still can’t find the IP address, for the WD My Cloud you can find the IP address in your settings, as per the instructions here.

In my case my NAS IP address was:

Mounting the NAS to the Raspberry Pi:

The first step to accessing the NAS from your Raspberry Pi is to mount the external HDD as a file system on the Raspberry Pi, this will allow you to view the NAS, as you would any directory on the Pi. This is pretty easy actually, as the CIFS (CIFS Common Internet File Share, a protocol dictating how different OS’ share files between them, including Windows and Linux) protocol takes care of everything.

First make a directory for the share:

# mkdir wdmycloud

Next mount the drive using the IP address and the Raspberry Pi directory you want to mount to:

# sudo mount -t cifs -o guest // /home/pi/wdmycloud
mount command

mount command

In this example, I am mounting the “Public” folder located on my NAS to the wdmycloud folder located on my Raspberry Pi.

The command syntax is:
mount -t <Mount Type> -o <Access> <SOURCE> <DESTINATION>

After executing the mount command, you should now be able to access the NAS file system as you would any other directory!

Automatically mount the NAS on power up:


Edit the FSTAB to mount your NAS automatically on power up:

To make the mount permanent, we need to add the NAS file system to the Raspberry Pi’s /etc/fstab file – the File System Table.

# sudo nano /etc/fstab
Edit the FSTAB file.

Edit the FSTAB file.

Add the NAS as a file system in the FSTAB file.

// /home/pi/wdmycloud cifs guest     0       0
Adding the NAS to the FSTAB.

Adding the NAS to the FSTAB.

You can see from the last line in the FSTAB file above, I have added the NAS as a file system in my FSTAB file.

This will automatically mount the NAS every time you power up your Raspberry Pi!

Testing the NAS is connected automatically on power up:

First step, reboot your Pi… :

# sudo reboot
Reboot your Pi.

Reboot your Pi.


Next step, check your NAS directory from the Pi:

# cd wdmycloud
# ls -larth
Confirm the share is working.

Confirm the share is working.

Success! The mount works, I can now access all of my pictures, music and movies from my Raspberry Pi. Everything is safely stored on my WD My Cloud, which keeps 2 copies of all of my data – so if anything goes wrong, I’ll always have my data backed up 😉


Making a Twitter-Bot on your Galileo or Raspberry Pi

Twitter have developed an API (Application Programming Interface) for their website, which makes it really easy to send and receive Tweets from your Raspberry Pi or Galileo! The Twitter API takes all of the hard work out of writing a program to interface with Twitter, There are several ways to access the Twitter API, the easiest of which (in my opinion anyway 🙂 is to use the Twython package of the Python language.

Differences between Raspberry Pi and the Galileo:

It’s really easy to install Twython on the Raspberry Pi, a little harder to install on the Galileo – so for that reason, I’ll show the step-by-step instructions from the Galileo install. The only difference is the Galileo doesn’t require you to use the sudo command as you already have root permissions set. For example, when editing a file with the Galileo you would use:

# nano my_file.txt

When editing the file on a Raspberry Pi you would use:

# sudo nano my_file.txt

Changing the date on the Galileo:

In order to install some of these packages, you’ll need to update the date and time of your Galileo – this isn’t automatically done when you connect to the internet as you may expect. If you don’t update your date and time, you’ll get SSL certification errors when you try to download the Twython package.

To change the date and time – use the following command in the format year-month-day hour:minute:second:

# date --set="2015-01-20 10:00:00"

Installing Twython (and other Python packages):

In order to connect our computer to Twitter, we’ll need to download some Python packages:

  1. setuptools – this will allow you to “Easily download, build, install, upgrade, and uninstall Python packages.”
  2. pip – this is a  Python package installer.
  3. twython – this is the package which will actually interface with Twitter.

Here are the commands to install these packages – remember if you’re on a Raspberry Pi you’ll need to put “sudo” in front of each command.

# apt-get install python-setuptools

Here’s what that looks like on your computer, when asked “do you want to continue” as per the picture below – enter “y”. to continue.

Package 1 of 3.

Installing the setuptools Python package.

Next, install the “pip” package:

# easy_install pip

Finally, using pip – install the Twython package:

# pip install twython


Creating a Twitter App:

To create a Twitter App, you’ll need to sign up to Twitter and register the account as an application. This is important as you’ll need to verify this App with Twitter every time you use it. The verification method Twitter uses is called OAuth 2.0 to verify your App, this means you’ll never have to supply your password to 3rd party App developers but it does make it a little harder to verify your App – the good news is, Twython and other API’s handle all the OAuth pain, all you need to do is register your App and save the information. Register your app here: Click on “Create New App” and enter your information.

Let the fun begin!

Creating an App.


Changing App permissions:

As this is your App and you’ll want to be able to play around with it – you can chance the App permissions to allow you to read, write and access your direct messages. You can change these permissions at a later stage.

Change permissions.

Change permissions.

Authorize your account:

You need to create your access token and access token secret before you use your App. Click on the “Keys and access tokens” tab. Click on the “Create my access token” button.

Generate your secret token.

Generate your secret token.

You should now have all 4 pieces of required information:

  1. Consumer Key (API key).
  2. Consumer Secret (API Secret).
  3. Access Token
  4. Access Token Secret.

Now it’s time to write some Python!

Writing the Python:

Creating a Python file:

On your Raspberry Pi or Galileo, create a file called “” using the following command:

# nano

Now paste in the Python code:

Twitter Authorization:

In order to send  a Tweet, you’ll need to send Twitter your OAuth information. This process is handled by the Twython package. Here we are creating 4 string objects and a Twython object called “twitter”. When we create the Twython object we are passing the 4 strings to it an arguments. These are the access keys you generated in the previous section.


Reading in a command line argument:

To make the code a bit more flexible, we can pass the text we want to Tweet into the Python script as a command line argument. This is done by using the system “argv” parameter. In our case – we only want to take the first 140 characters of this text, as this is the character limit set by Twitter for each Tweet. We do that by using the command:


Executing the code:

You can execute the code from the command line like so:

# python "Hello world."

Here’s what that looks like on the Galileo:

First Tweet!

Hello World!

Then check your Twitter Bot!

Installing Debian on the Intel Galileo

Some of the guys in the Intel communities forum have done a great job of porting the Debian Linux distribution to the Intel Galileo. If you’re not familiar with the Debian, it is an open source operating system used on Linux systems. This is great news for the Galileo as it makes it very easy to port any Raspberry Pi based projects directly to the Galileo!

Debian was ported to the Galileo by StuartAnderson, who posted on the Intel Communities page.

You can download the Linux image from here!

What you’ll need:

  1. Galileo.
  2. SD card.
  3. Serial cable.

Downloading the Galileo Debian disk image:

You can find the Galileo Debian image here. Click on the “SD card image” file. This will take some time as the file is 900MB. While the disk image is downloading, we can prepare the SD card.

Formatting your SD Card:

If your SD card has been used to write a Linux image before (used for a Raspberry Pi or Galileo Little Linux image), Windows can have trouble formatting the SD card. This is because Windows can only see the FAT32 partition and won’t see the much bigger Linux partition.

For this reason I used the SDformatter program from the SD Association, you can find it here (Click on Windows or Mac on the left).

You can see from the picture, I’m using an 8GB SD card (7.4GB shown but it’s 8GB, it’s normal for SD cards to be about 10% smaller than the advertised disk size), in my case Windows format only saw 56MB of the disk and ignored the remaining 7.35GB! However with the SD Formater program the full 7.4GB can be seen and will be formatted. Click on the Format button to start the process.


Using Rawrite32 to write the Galileo Debian disk image.

Writing the Debian Image to an SD card:

To write the Linux Debian image to the SD card you’ll need to download a disk image tool, as recommended by the original creators, I used the Rawrite32 Disk Image Tool software, found here.

After you’ve downloaded the Debian image, open Rawrite32 and select the Debian image to write.

Whoop - Debian!!!

Writing the Galileo Debian image.

You’ll see a pretty scary warning like this, but as long as you’re sure there’s no important data on the SD card that you need, go ahead and click “Yes”!

Alt text for this picture.

Click OK to start writing the Disk Image.

Once the Disk Image tool has finished writing – you’ll see something like this:


Finished writing the image.

You can see from the image above, the image file takes up 900MB, less than 1GB of my 7.4GB SD card. This means that the remaining 6.5GB of space on the SD card will not be available to me when I try to the Linux image. In order to make the remaining 6.5GB of space available, we need to grow the size of our partition on the SD card.

Now that we have a Debian Linux system available – this can be done from the Galileo itself!

Expanding the filesystem:

To expand the filesystem, first we need to expand the partition, then expand the file system to fill that partition.

When formatting the SD card, we could see that the 8GB SD card had a size of 7.4GB, so I’m going to expand my partition to 7.4GB just to be on the safe side.

Disk size before partitioning:

Not enough room!

Before resizing the filesystem.

Expanding the partition:

To resize the partition, we are going to use a program called Parted. This can be used to create, resize, move and destroy disk partitions. The first command used is the resizepart command.

# parted /dev/mmcblk0 resizepart 2

Select “y” to continue and then enter the size of the partition you want to create, here’s where I entered the 7.4GB (which works out as 7400MB).

# parted /dev/mmcblk0 resizepart 2

# parted /dev/mmcblk0 resizepart 2

You’ll need to reboot here!

# reboot

Expanding the filesystem:

When the partition has been resized we create a journal inode.

# tune2fs -j /dev/mmcblk0p2
# tune2fs -j /dev/mmcblk0p2

# tune2fs -j /dev/mmcblk0p2

You’ll need to reboot here!

# reboot

When the reboot has finished, finish by expanding the filesystem to the partition size.

# resize2fs /dev/mmcblk0p2
# resize2fs /dev/mmcblk0p2

# resize2fs /dev/mmcblk0p2

The resize2fs will expand the filesystem to fill the partition, this will take about 10 minutes but when it’s finished you should see a message like this, above.

Disk size after expanding the filesystem:

Loads of room!

Disk size after expanding the partition.

From the above image you can see, we’ve grown the filesystem to fill the partition, so we now have a total disk size of 6.8GB, with 6GB available to play with!

Now that the filesystem is expanded, it’s a good idea to update the system.

apt-get update && apt-get dist-upgrade

The update and upgrade will take about 10 minutes, then you can have fun playing with your new Debian computer.

© 2017 Allyn H

Theme by Anders NorenUp ↑