Knowledge is power. We love to share it.

News related to Mono products, services and latest developments in our community.

mario

Building an augmented reality app for iPhone with MonoTouch

08/28/2012Categories: Mobile

Introduction

A few months ago I received a Mac Mini and an iPhone 4S and got task to develop an augmented reality (AR) application for iOS, focusing on location services and GPS positioning.

A basic idea was to create an application that could display Points of Interest (POI) locations on the screen while video camera is active, and continously track POI locations as they change. Apple's push notification infrastructure was to be used to receive information about POI location change. The final goal is to be able to get a visual indication of the position of other people running the application on their mobile devices overlayed on the iPhone's camera output. To be able to do this, our mobile devices exchange information with a server that runs on top of the ASP.NET Web API framework. To make things more interesting, the location information can be sent from Android devices - one of my posts will cover basics of programming with Mono for Android. Please be aware of the fact that our company is not related in any way with the Mono Project - Mono Software is just a happy user of some of their products. 
As AR development was a new experience for me, I figured that it would be a good idea to share my newly acquired knowledge with the rest of our community in a series of blog posts on our site. There is simply too much ground to cover in a single blog post, so please stay tuned for next articles in this series.

I set up my development environment using all latest tools, including Xamarin's MonoDevelop 3.0.3 with MonoTouch 5.1 framework and XCode 4.3.3 running on Mac OSX 10.7.4 Lion. My iPhone runs iOS 5.1.1.

We are using C# a a primary development language in Mono Software, and although Objective-C was a viable option, we tried MonoTouch, quickly fell in love with it and never looked back. 

Preparing the Environment

Since I was working in a VS.NET / Windows environments for years now, developing on Mac was quite annoying, at least for the first couple of days. Different shortcuts, OS layout, installation processes – all that needs some time to settle down with a new user.

Installing tools on Mac

There are a lot of information on installing these tools, but things didn't work as described on many occasions, so I’ll try to give you a few hints to ease the process.

There is no good way to test sensor capabilities in the iOS simulator provided with MonoDevelop, and some of them are not testable at all, so we used an iPhone 4S 16GB and upgraded it to iOS 5.1 right after unpacking. During the installation process your AppleId must be created, and (if you want to develop for iPhone), an iOS developer account needs to be created at the same time. This account will be used for installing XCode.

The next step was to install MonoDevelop and MonoTouch, and it didn't go smoothly as expected. My version of MonoDevelop requires XCode 4.3 (I’ll describe using XCode with MonoDevelop later). So I went to the Apple store and downloaded the latest XCode. To my surprise, I was greeted with the message that XCode 4.3. requires Mac OS X 10.7 Lion. It took some time, but it payed of, and I installed OS X Lion, XCode 4.3.2 and MonoDevelop 3.0.3.2, in that order.

Installing Certificates

To be able to deploy application to your iPhone, besides developer certificate and key, you also need certificate and provisioning profile for the application. Our application will  use push notifications, so we need a push provisioning profile.

I found a great post with step by step instructions for installing these certificates, profiles and configuring XCode for deployment on the iPhone, provided by Matthijs Hollema: http://www.raywenderlich.com/3443/apple-push-notification-services-tutorial-part-12 . Since I wanted to build MVC 4 WebApi application for sending push notifications to APNS (Apple Push Notification Server), some time was spent configuring Windows to allow me to send messages to APNS. At the end,  it was all fairly simple. As Matthiys said in his post, you need to pack your certificate and key in a p12 file and then import that file in Windows certificate store.

Creating an iOS Application

We will be using MonoDevelop for writing application code, building and deploying, and XCode will be used for drawing the application user interface.

View And Controllers - Main View

We are developing an iOS application in MonoDevelop using the MVC approach, which means that for every screen we want to display, a View part and a Controller part are created. Our view part contains layout data and is designed in XCode, while Controller is a class which inherits from UIViewController and contains all logic related to the view. We use controller to show, update and remove views. Every view can be container for other subviews, and every subview can be container for other subviews,  and so on. So when we are talking about controls we are thinking of subviews contained within another view.

For simplicity's sake, our application will have only one main view and only one controller. Since we will add all controls programmatically, we don’t need to do anything with our view using the XCode designer. Main view will display POI locations with camera preview in the background. It will also contain a button for connecting to our server application and labels for displaying number of POIs monitored. We also want to provide the ability to lock user's location and enable or disable each of his sensors. Here is how this view should look like in the landscape mode:

Creating the solution

We will start by creating a new Single View Application project using MonoDevelop and rename the default controller to iPhoneAppViewController.cs (and .xib). This class will contain all our application logic. I refactored a big part of this logic into helper classes that I will describe in my other posts. We will use all those helper classes in this project - they are located in the Classes folder. Note that a zip file that is attached to this post contains a complete solution that can be used in your scenarios. Here is how it looks in the Solution Explorer:

Along with our helper classes and the main application controller there are folders Controls and Settings. Controls folder holds two classes with custom controls we will use in our view and the Settings folder contains ApplicationSettings.xml and ApplicationSettingsGenerator.tt - the XML file contains settinsg for our application and the T4 template file is used to generate C# representation of the XML file.

Configuring the solution

After setting the project properties, we need to set a few configuration parameters in the ApplicationSettings.xml: sensor update intervals, field of view settings and the application server URL that will be used to exchange data with the server - more on that in my next post. After you finish editing the XML file, just open the T4 template, save it and the  ApplicationSettings class will be (re)generated. Here is the sample configuration I used:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <LocationMonitor>
        <ApplicationPurpose>SampleApp location monitor</ApplicationPurpose>
    </LocationMonitor>
    <MotionMonitor>
        <GyroUpdateInterval>0.1</GyroUpdateInterval>
        <AccelerometerUpdateInterval>0.1</AccelerometerUpdateInterval>
        <DeviceMotionUpdateInterval>0.1</DeviceMotionUpdateInterval>
    </MotionMonitor>
    <!-- Landscape mode
    <FOV>
        <!-- For iPhone in radians (float)2.0 * (float)Math.Atan (58.5 / 2.0 / 56.5);
        // For iPhone 4S in radians (float)2.0 * (float)Math.Atan (4.592 / (2 * 4.28)); -->
        <Horizontal>0.98475913484564</Horizontal>
        <!-- For iPhone in radians (float)2.0 *  (float)Math.Atan (21.5 / 2.0 / 56.5);
        // For iPhone 4S in radians (float)2.0 *  (float)Math.Atan (3.450 / (2 * 4.28)); -->
        <Vertical>0.76624413062396</Vertical>
    </FOV>
    <Notification>
        <ApplicationServerUrl>
            <Subscribe>http://192.168.0.193/WebApi/api/Subscribe<;/Subscribe>
        </ApplicationServerUrl>
    </Notification>
</Configuration>

Implementing the main controller

If we look in the iPhoneApplicationViewController.cs, the first two regions are Fields region with declarations of the fields used in our controller and the Controls region with declarations of UI controls mapped to the view.

Dependency injection is used to pass the reference to the application delegate to the main controller:
#region Constructor
public iPhoneAppViewController (AppDelegate appDelegate) : base ("iPhoneAppViewController", null)
{
    app = appDelegate;
    HandleMotionEvents ();
}
#endregion
Constructor is also used to hook up to the motion events of a device. This way, all changes of yaw and roll values will trigger the update of the main view.
void HandleMotionEvents ()
{
    app.DeviceListenerShared.YawChanged += yaw => {
        DrawFriends ();
        UpdateDebugInfo ();
        UpdateSensorsState ();
    };
    app.DeviceListenerShared.RollChanged += roll => {
        DrawFriends ();
        UpdateDebugInfo ();
        UpdateSensorsState ();
    };
}
We are overriding different event methods of a view and use them to update the appearance of our controls. Some of these methods are ViewDidLoad() where controls are initialized, status bar is hidden and the device is subscribed for notifications. DidRotate() contains code for setting the heading orientation and view layout updates. All of them basically contains calls to other helper methods with and no real logic, and the complete code can be found in the Overrides region of iPhoneAppViewController.cs in the attached solution.
Going further, when a device changes its orientation, the location manager HeadingOrientation property must be updated, and SetHeadingOrientation() is called to perform that task:
private void SetHeadingOrientation (UIInterfaceOrientation toInterfaceOrientation)
{
    if (app.DeviceListenerShared != null)
        return;
    if (toInterfaceOrientation.Equals (UIInterfaceOrientation.LandscapeLeft))
        app.DeviceListenerShared.LocationManager.HeadingOrientation = CLDeviceOrientation.LandscapeLeft;
    else if (toInterfaceOrientation.Equals (UIInterfaceOrientation.LandscapeRight))
        app.DeviceListenerShared.LocationManager.HeadingOrientation = CLDeviceOrientation.LandscapeRight;
    else if (toInterfaceOrientation.Equals (UIInterfaceOrientation.Portrait))
        app.DeviceListenerShared.LocationManager.HeadingOrientation = CLDeviceOrientation.Portrait;
    else if (toInterfaceOrientation.Equals (UIInterfaceOrientation.PortraitUpsideDown))
        app.DeviceListenerShared.LocationManager.HeadingOrientation = CLDeviceOrientation.PortraitUpsideDown;
}
The DrawPOIs() method calculates the coordinates for each POI, alter coordinates for the POIs that are already on the screen, tries to find the existing control for the POI in the current view and if there is one updates its location, or otherwise create a new control for the POI. The same method finds the POI that is closest to the center of the screen to display the information about it.
public void DrawPOIs ()
{
    btnData.SetTitle (String.Format ("POIs: {0}", app.LocationsToDraw.Count.ToString ()), UIControlState.Normal);
    var cameraLocation = new CLLocationCoordinate2D (app.DeviceListenerShared.Latitude, app.DeviceListenerShared.Longitude);
    MKMapPoint currentCoord = MKMapPoint.FromCoordinate (cameraLocation);
    var friends = app.LocationsToDraw;
    int nearestX = 0;
    bool nearestChanged = false;
    bool nearestOffScreen = false;
    Circle nearestControl = null;
    foreach (var friend in friends) {
        Circle btn = null;
        btn = this.View.ViewWithTag (friend.Id) as Circle;
        Point screenCoord = Helper.GetScreenPoint (
            friend,
            currentCoord, (float)app.DeviceListenerShared.Altitude,
            (float)app.DeviceListenerShared.Yaw,
            (float)app.DeviceListenerShared.Roll,
            (int)this.View.Bounds.Height, (int)this.View.Bounds.Width,
            (float)ApplicationSettings.FOV.Horizontal,
            (float)ApplicationSettings.FOV.Vertical,
            app.DeviceListenerShared.LocationManager.HeadingOrientation
        );
 
        //Find nearest POI
        nearestChanged = false;
        if (Math.Abs (240 - screenCoord.X) < 25
            && Math.Abs (240 - screenCoord.X) < Math.Abs (240 - nearestX)) {
            nearestX = screenCoord.X;
            nearestPOI = friend;
            nearestControl = btn;
            nearestChanged = true;
        }
 
        UIColor color = GetAlwaysOnScreenAndColor (ref screenCoord, ref nearestChanged, ref nearestOffScreen);
        int circleSize = 10;
        if (btn == null) {
            btn = new Circle (screenCoord.X, screenCoord.Y, circleSize, 2, color);
            btn.Tag = friend.Id;
            btn.SetTitle (friend.Id.ToString (), UIControlState.Normal);
            this.View.AddSubview (btn);
        } else {
            btn.MoveTo (screenCoord.X, screenCoord.Y, color);
        }
    }
 
    if (nearestControl != null) {
        if (nearestOffScreen)
            nearestControl.SetTitleColor (UIColor.Yellow, UIControlState.Normal);
        else
            nearestControl.SetTitleColor (UIColor.Blue, UIControlState.Normal);
    }
}
Main controller additionally contains logic for subscribing to notifications. This is done by sending a device token to our .NET WebAPI service (again, I will present more info on the server-side logic in one of the next articles). This token is used by APNS to identify iOS devices. Subscribing to the WebAPI service is done by using AsyncCallback methods to keep UI more responsive. We are using System.Net.WebRequest to send JSON data with the token to the server.
private void SendDeviceToken ()
{
 
    if (!String.IsNullOrEmpty (NSUserDefaults.StandardUserDefaults.StringForKey ("deviceToken"))) {
         
        string lastDeviceToken = NSUserDefaults.StandardUserDefaults.StringForKey ("deviceToken")
            .Replace (" ", String.Empty).Replace ("<", String.Empty).Replace (">", String.Empty);
        string url = ApplicationSettings.Notification.ApplicationServerUrl.Subscribe;
        webRequest = WebRequest.Create (url) as WebRequest;
         
        System.Json.JsonObject jsonNotif = new System.Json.JsonObject ()
        {{"Token", lastDeviceToken}};
        messageBody = jsonNotif.ToString ();
        webRequest.ContentLength = messageBody.Length;
        webRequest.Method = "POST";
        webRequest.ContentType = "application/json";
        btnConnect.SetTitle ("Connecting...", UIControlState.Normal);
        btnConnect.SetTitleColor (UIColor.Yellow, UIControlState.Normal);
        Console.WriteLine ("Sending token...");
         
        webRequest.BeginGetRequestStream (new AsyncCallback (GetRequestStreamResponse), null);
    }
}
 
private void SendDeviceTokenResponse (IAsyncResult ar)
{
    webRequest.EndGetResponse (ar);
    if (ar.IsCompleted) {
        Console.WriteLine ("Token sent.");
    } else {
        Console.WriteLine ("Token not sent!");
    }
 
}
 
private void GetRequestStreamResponse (IAsyncResult ar)
{
    Console.WriteLine ("Waiting response...");
    StreamWriter stOut = new StreamWriter (webRequest.EndGetRequestStream (ar), System.Text.Encoding.ASCII);
    stOut.Write (messageBody);
    stOut.Close ();
    webRequest.BeginGetResponse (new AsyncCallback (SendDeviceTokenResponse), null);
    Console.WriteLine ("Token sending in progress...");
}

Using the sample application

Sample application should be pretty simple and easy to use. It is designed to support landscape interface orientation as a primary working mode, but will also work in the portrait mode. After application starts, its main (and only) view is displayed.
I have placed sensor-related buttons in the top left part of the main window. These buttons have three states: red color is used if a sensor is unavailable, yellow color is used if a sensor is not active but available, and green color is used if a sensor is active. Sensors are (from left to right): device motion sensor, magnetometer, gyroscope, accelerometer and heading. Clicking on the each button will toggle its state.

The bottom section of the view displays the current server connection state, a button for locking the device location and a button with number of POIs monitored. You can click on this button for testing purposes, and Mono office location will be added as a POI.

The right section of the screen displays debug information. Touching this label will toggle its visibility state. Debug information include (in this order): latitude, longitude, altitude, yaw, roll, heading accuracy, horizontal accuracy, vertical accuracy, sensors state, reference frame and, if a POI is focused, distance to the POI and its altitude.
Complete solution is attached in a zip file, feel free to use it anyway you like.
Rated 4.40, 10 vote(s). 
You sample has an issue when the camera crosses north. If you added more than one POI (I put 10 all around me), the POIs that are to the north of me will jump from one side of the screen to the other as I cross due north. I am going thru your GetScreenPoint function to try to figure it out, but didn't know if you had already solved this issue.