The Fat Lady Didn't Sing

On August 4, Clarke from the MobileMe Support Specialist Team made contact via email to ask whether I was still interested in resolving this case. Although pleasantly surprised by the renewal of interest on the part of Apple initially I ignored the mail fearful of another waste of time with an agent who knows nothing. However, when Clarke again contacted me on August 19th I thought I'd give him marks for persistence and respond. MobileMe Support must have more bandwidth now - maybe because everyone has canceled already? :-)

Responding to Clarke turned out to be a good move as he reactivated my expired MobileMe account in order to continue to debug the problem. He also gave me a program called CaptureData which grabs all sorts of data from your machines (including packet traces) into a dmg which you send to Apple. Given those carrots I've sent him 7.6Mb of data to chew on! Hope he finds something in there.

Since the initial problems earlier this year things have moved on a bit. We have moved from Leopard 10.5.6 to 10.5.8, with Snow Leopard due out tomorrow! The time capsule firmware has also been bumped from 7.4.1 to 7.4.2. Screen Sharing from within iChat is another useful alternative to Back To My Mac in certain circumstances also. With my MobileMe account re-enabled I re-ran my matrix of tests. Turns out Apple have made quite some steps forward and most things actually "just work" now! Screen Sharing works all over. Finally.

The only remaining problems are file sharing from site A to the Time Capsule at site B and file sharing at site A being inaccessible to site B. I didn't test at sites C, D and E yet, until I see what Clarke comes back with. Let's hope he's Superman!

Trial has run out

Game over. No callback. No resolution.

Waiting

Status is that AppleCare have given up. Have been waiting for a MobileMe support specialist to call me since April 28.

Lost the trail in IPv6

The previous post hinted that communication between BTMM hosts is done via IPv6. These requests were either not getting through to the target machines, or their replies weren't. In the Time Capsule configuration there is an IPv6 configuration pane in the Advanced tab that may hold the answers.

The explanation text is a bit "Microsoft help" and the manual useless but you can deduce what they mean. The default mode is link-local which means that IPv6 traffic is restricted to the LAN side of the router. There are two other options: Node and Tunnel. I can assume Node gives the router a presence on the network, with it's own address? Whereas tunnel makes it act as a bridge? Or tunnelling IPv6 over IPv4? Is it a tunnel broker? Not sure. Don't know how to find out more. If it was providing a tunnel would I be able to access the Time Capsule itself for file sharing? If it was a node would I be able to access the machines behind it? Stuck.

I say stuck because I tried the various configurations of node and tunnel between my two Time Capsules and everything went a bit screwy on my network. It was highly confusing how anything was communicating...4, 6, both? I need to understand this more to make sense of it. In tunnel mode an IPv6 firewall tab appears apparently, allowing incoming IPv6 connections.

So this is as far as I got. I had no more settings to twiddle to try to get BTMM to work. Ironically, I think it would be more likely to work without Apple routers involved. Something with a more transparent IPv6 behaviour and UPnP perhaps?

So I proceeded to create my own, more traditional, approach using port forwarding, NAT and VNC with DynDNS.org helping me out.

Digging deeper

From the previous post, you now know your service names. You now know how to construct them yourself so you don't even need to look them up, but it's a good check to see that everything you expect is there.

By passing the -G flag to dns-sd you can get more detail on individual hosts.

$ dns-sd -G v4v6 PowerBookG4.graterfamily.members.mac.com
Timestamp     A/R Flags if Hostname                  Address                                      TTL
20:32:44.971  Add     3  0 PowerBookG4.graterfamily.members.mac.com. FD5E:D071:42E8:F66E:020A:95FF:FED4:1BEE%<0>  152
20:32:44.972  Add     2  0 PowerBookG4.graterfamily.members.mac.com. 0.0.0.0                                      0   No Such Record
^C

$ dns-sd -G v4v6 WirelessBackup.graterfamily.members.mac.com
Timestamp     A/R Flags if Hostname                  Address                                      TTL
20:33:29.226  Add     3  0 WirelessBackup.graterfamily.members.mac.com. FD64:355A:6320:A136:021F:F3FF:FEC8:1639%<0>  152
20:33:29.227  Add     2  0 WirelessBackup.graterfamily.members.mac.com. 124.155.35.249                               152
^C

Here you can see how you have to ^C out of the command as it hangs looking for changes. Above you can see how the PowerBookG4 on the internal network doesn't have an IPv4 address. The WirelessBackup machine does as it's a Time Capsule operating as a router. So the IPv4 address shown is the public address of that LAN - common to all machines behind (and including) the Time Capsule.

You notice how both machines have an IPv6 address. There are enough IPv6 addresses for everyone so in the 6 world there is no more need for NAT and sharing public IP addresses other than security. This address gets direct to your host. Apple are quite forward in their IPv6 support.

Dynamic DNS entries within MobileMe

As you may have twigged from the post below, when you turn on Back To My Mac in the MobileMe preference pane (without errors) your hosts are registered in DNS by MobileMe.

The format is hostname.membername.members.mac.com. where hostname is the name of your machine shown at the top of the Sharing preference pane and membername is your MobileMe login id. This is achieved with (wide-area) Bonjour.

To query for your entries you need to use the dns-sd utility (DNS Service Discovery).

$ dns-sd -B _afpovertcp._tcp
Browsing for _afpovertcp._tcp
Timestamp     A/R Flags if Domain                    Service Type              Instance Name
20:32:26.788  Add     3  0 graterfamily.members.mac.com. _afpovertcp._tcp.         WirelessBackup
20:32:26.789  Add     3  0 graterfamily.members.mac.com. _afpovertcp._tcp.         TimeCapsule
20:32:26.789  Add     3  0 graterfamily.members.mac.com. _afpovertcp._tcp.         WirelessBackup
20:32:26.789  Add     3  5 local.                    _afpovertcp._tcp.         hidden
20:32:26.789  Add     2  5 local.                    _afpovertcp._tcp.         TimeCapsule

Here you can see the both the remote entries and the local (LAN) services being offered for AFP running over TCP (Apple File Sharing). You can see how the machine hidden is only offering services locally (it has file sharing turned on but Back To My Mac off). In this example I am querying for file sharing services but you may (also) want to query for VNC (screen sharing) services by specifying _rfb (remote frame buffer) in place of the _afpovertcp._tcp above or as:

dns-sd -B _rfb graterfamily.members.mac.com

There is a list of services you can browse for and an explanation of the DNS entry format for Bonjour from Apple.

Snoop Dogg

This dog has got way more Mac knowledge than I gave rappers credit for. His suggestion for using OpenDNS servers to correctly resolve SOA queries gets a lot of coverage on the forums and solves issues for many people. Unfortunately, not me! :-)

I tried what he suggests but my ISP's at both ends of my connection seem to resolve SOA queries without issues.

dig bob.members.mac.com soa @202.224.32.1

; <<>> DiG 9.4.2-P2 <<>> bob.members.mac.com soa @202.224.32.1
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54024
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;bob.members.mac.com.        IN    SOA

;; ANSWER SECTION:
bob.members.mac.com.    7200    IN    SOA    pm-members.mac.com. postmaster.mac.com ...

;; AUTHORITY SECTION:
bob.members.mac.com.    86400    IN    NS    pm-members.mac.com.

;; ADDITIONAL SECTION:
pm-members.mac.com.    1612    IN    A    17.250.248.161

;; Query time: 257 msec
;; SERVER: 202.224.32.1#53(202.224.32.1)
;; WHEN: Thu May  7 15:46:59 2009
;; MSG SIZE  rcvd: 125


dig bob.members.mac.com soa @165.21.100.88

; <<>> DiG 9.4.2-P2 <<>> bob.members.mac.com soa @165.21.100.88
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21400
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;bob.members.mac.com.        IN    SOA

;; ANSWER SECTION:
bob.members.mac.com.    7200    IN    SOA    pm-members.mac.com. postmaster.mac.com ...

;; Query time: 212 msec
;; SERVER: 165.21.100.88#53(165.21.100.88)
;; WHEN: Thu May  7 15:47:02 2009
;; MSG SIZE  rcvd: 95

When silence isn't golden

It's been a week now, so I think we can safely say that MobileMe support have given up. If AppleCare gave up I wasn't hopeful for the MobileMe guys. In actual fact, I have also given up! In the last week I haven't been idle. I've done a lot of research on the web but still not got a BTMM solution. However, I went back to first principles and got a solution that does what I wanted BTMM to do for me - it's much easier! Over the next few posts I'll outline what I found out about BTMM and tools for debugging it. I'll finish by outlining my BTMM solution.

Nothing in my inbox

No surprises. I've begun another push at trying to help myself again via prodigious Googling:

• I've configured and rebooted all machines & routers to be in JST - I read something on the net about Kerberos authentication being fussy about differences so maybe there is something in this after all?
• I've been debugging the DynDNS registrations in MobileMe using the dns-sd command. When (if?) I work out what's useful about it I'll post.
• I found something about IPv6 on the net which tallies with the output of the above command. The machines get registered at Apple with an IPv6 address. But the Time Capsule defaults to having IPv6 setup as "link-local only" (which I think means LAN only). I tried setting this to Node and Tunnel on my end but think it needs to be Tunnel on the remote end. I've put in a request for that. Also, this tallies with an NTT support page which says that to receive their remote support for the B-Flets service we have you need a router capable of IPv6 tunnelling. I hope this is it!

Back To MobileMe Support

info: Hi, my name is Kelvin J. Welcome to Apple!
John Grater: Hi
Kelvin J: Hello John
John Grater: I've got an ongoing issue with Back To My Mac
John Grater: I started here last Monday
Kelvin J: Ok.
John Grater: Then got bounced to AppleCare
John Grater: Luckily I have an AppleCare contract
John Grater: They escalated to engineering
John Grater: via two product specialists (lastly Karl)
John Grater: Engineering came back that it's up to MobileMe support
John Grater: So I'm back
John Grater: Can someone give me a call to go through this?
John Grater: I have an AppleCare case number if you can read the notes from there
Kelvin J: Let me have the case number.
John Grater: Basically the problem is that I can see all my machines in the Finder but get "Connection Failed" whenever I try to connect
Kelvin J: You are on a broadband connection on your end correct?
John Grater: Yes. The other end is on 100Mbps fibre
John Grater: This was escalated before to Craig on your team who bounced me to AppleCare initially
John Grater: My initial contact was with Chris W
Kelvin J: Great. Right now I am just reviewing your last chat and pulling up the Apple Care case. Would you mind standing by while I review your history.
John Grater: I need someone to check the dynamic DNS entries for my machines in MobileMe
John Grater: and then to debug the resulting log messages on my Time Capsule
John Grater: Ok
Kelvin J: Maybe we might have an update on the escalation.
Kelvin J: Do you have Double NAT setup on your routers at home?
John Grater: No
John Grater: I checked it, and all the other web-based troubleshooting guides
Kelvin J: Great. This way way I can eliminate unnecessary repeated steps.
John Grater: With all due respect, this has gone all the way to engineering - so I think we need to think about trying to debug the deeper issue
Kelvin J: Thank you for waiting. I'll be with you in just a moment.
Kelvin J: I am tend to agree since viewing your history with previous chats, unfortunately we don't have any available at this time. They will be back online in 5.5 hours.
John Grater: That
John Grater: That's 9pm my time - I can still be up
John Grater: Can you have somebody call me?
Kelvin J: Not from here, but if you chat back in the can chat with the escalation time they have capabilities we do not.
John Grater: But how can I get directly through to someone who can help me?
Kelvin J: Standby let me check on that, because as far as I know they rely on the chat system for the bulk of issue with MobileMe but with your particular situations requires specific attention.
John Grater: Yes please. Also, as it's going into my night time I cannot spend a long time trying to get through to the right person
Kelvin J: Believe me I understand.
John Grater: Thanks
Kelvin J: I'm sorry for the delay. I'll be right with you.
John Grater: No problem
Kelvin J: Ok John this what I can do.
Kelvin J: I am going to send your information including your phone number requesting for a call back for you.
John Grater: That is correct
John Grater: So any time before 11pm is ok
Kelvin J: Great. More and likely they will send you a e-mail stating a time for contact to ensure availability.
John Grater: That's great
Kelvin J: Ok John I think I have everything I need to get you taken care of here and I hope our engineers get this finally resolved for you.
John Grater: Thanks
John Grater: Have a good day
Kelvin J: You too, John and thank you for chatting with us. We value your feedback. Please click the blue "Close" button at top left to answer a few questions about your experience with us today.
John Grater: Ok. Bye.

Time Zone rubbish

Tried changing the time zone on my Mac to Japan time but, of course, no different. All the machines are NTP sync'ed so the UTC times are all in check, which is what I think engineering were referring to, rather than the time zone.

Response from engineering

Karl called up with the response from engineering. They replied with a generic check list but we went through it anyway:

• All machines running 10.5.6? Check!
  
• None of the machines are asleep? Check!
• The diagnostics in the Back To My Mac panel report no problems? Check!
• The firmware is up-to-date on all Time Capsules? Check!

The other suggestion was that is could be due to my working across time zones. Karl mentioned that even connecting within the same building, if the time is off by only 20 mins then machines wouldn't connect. I'd seen this on the web before but found it hard to believe as surely the system would be using UTC time around the world.

Engineering said there was no more they could do from their side and so if the time suggestion doesn't help I have to go back to MobileMe support. To get them to take me more seriously I should mention that I spoke with Karl of the product specialist department and have already escalated to engineering who put it back on MobileMe.

So let's try changing time zone...

Got a call!

Karl called me back after my voice mail message which was nice. He explained that there is nothing he can do to push engineering until they respond (an engineer gets assigned to a problem and therefore we get a name & number to chase).

He agreed it's taking a little longer than usual (2 days) but promised to call me back as soon as he got a response from engineering.

In turn I agreed to leave him alone until then as there was nothing more he could do to push the issue.

More voicemails

Left voicemails for Ken and Karl again to get back to me.

Waiting

Good to know Karl is checking his voice mail messages and not just ignoring me:

"I received your call this morning and am aware that you are awaiting my response. I apologise for the delay but i am still awaiting the Engineering response. This can take some time and I appreciate your patience. I will endeavour to inform you of their response as soon as it comes through." (via email)

Give them a day

I decided to let the guys have Thursday to think it over and chat amongst themselves, and to use the day to try to recover my sanity.

Where's Ken?

Having not been able to get back in touch with Ken I called the general AppleCare number again and, after explaining the situation to the initial representative, got through to another technical specialist called Karl. He mentioned Ken was busy at the moment but had just tried calling me (perhaps we overlapped?). After looking through the notes of the call and hearing my summary Karl agreed to escalate the problem to engineering. They should be able to offer more debugging steps etc. Unfortunately the AppleCare guys cannot access any MobileMe support so it's not possible for them to check the dynamic DNS entries for my hosts within MobileMe.

So now we wait for a response from engineering...

Left a voicemail for Ken

Trouble with support guys is they are always on the phone! Hopefully he'll get back to me.

On to paid support

So now I invoke my AppleCare contract, purchased previously at not inconsiderable expense. I'm expecting much better service considering.

Got through (and accepted) into the AppleCare system with a case number and spoke to a guy called Ken who took me through all the same things as the MobileMe support guy (after I explained they referred me to him). Thankfully Ken agreed to call me back to save my phone bill which was helpful. Realising this was also going nowhere I again asked for escalation and got put through to a technical specialist called Kenneth (the same guy just putting on a different voice?).

Kenneth looked through the call notes and realised there wasn't much more he could suggest. So he asked for some time to look into the problem. I was suspicious of never being able to get back to him so got his extension number.

Brush Off

Recieved the promised email from Craig, a MobileMe Support Specialist.

"I understand that you are trying to connect to your Time Capsule's drive remotely using the Back to My Mac service. This is actually supported by the Wireless Multimedia team with AppleCare."

This was somewhat surprising seeing as Back To My Mac is part of MobileMe. But luckily, I have an AppleCare support contract. So it's over to them...

First Contact

Had a chat with a member of the MobileMe support team and described my problems. They were obviously going through items on the troubleshooting webpages as I'd already been through them all. After some time (an hour?) the representative agreed to escalate my call. I was promised an email within 24 hours to establish contact with a more senior technical assistant.

"info: Hi, my name is Chris W. Welcome to Apple!
Chris W: Hi John, how are you today?
John Grater: Hi, I'm stuck with Back To My Mac
Chris W: Ok, have you contacted us about this before?
John Grater: I have everything set-up with a Time Capsule at both ends
John Grater: No
John Grater: Both TC's are connected to the internet via bridges
John Grater: I am attempting to connect to the remote TC from this MacBook
John Grater: I see the remote TC in the Finder but every time I try to connect it says Connection Failed
John Grater: I have tried deleting all keychain items
John Grater: I have no diagnostic messages in the MobileMe System preferences pane
John Grater: It's an all Apple setup end-to-end
Chris W: I understand. What routers are you using?
John Grater: Time Capsules both ends doing PPPoE
Chris W: Alright, can you give me just a moment to look into this for you?
John Grater: Sure. Thanks.
Chris W: You're very welcome John, and I'll be right back
Chris W: Have you recently installed any new routers or anything onto your systems?
John Grater: This is my first time with BTMM. I've never had it working.
Chris W: I understand. I was asking about the routers because they have to be correctly configured as well
John Grater: The routers are both Time Capsules. I followed all the instructions and have been through the troubleshooting steps.
John Grater: Can you check the DynDNS entries in MobileMe for my account?
Chris W: Ok, give me just a moment to look into this for you.
Chris W: I'm sorry for the wait John, I'm looking over some resources for you.
Chris W: I was also looking over time capsule, and it isn't a router. That could be part of the issue.
Chris W: Do you know how to preform a trace route?
John Grater: Yes, I've done that - it's not double NAT'ing. Entry 2 is a real address.
John Grater: TC is a fully featured router by the way. It's an Airport Extreme base station with a disk in
Chris W: Unfortunately, it's not designed to be used for BTMM quite like you're trying to use it as.
John Grater: It has a BTMM configuration pane in it
Chris W: Ok, I understand, and if you don't mind I will look into this further for you. I apologize for any inconvenience that this has caused you.
John Grater: I don't mean to be rude but I think I need to talk with a BTMM and networking expert
John Grater: I'm an IT professional myself and so this isn't just a simple configuration issue
Chris W: I understand, and am trying to locate information for you that will be of use to resolve this issue. I would like to help get this issue resolved asap for you and want to cover all of the resources available to us first.
Chris W: John, do you mind if I send you a link to make sure that you've covered this already?
John Grater: Go for it
Chris W: http://support.apple.com/kb/TS1626
John Grater: Been there. Don't have any of those messages.
Chris W: Ok, then you're right. What I'll go ahead and do is escalate this matter to one of our BTMM Specialists. Is that okay with you?
John Grater: That's great. Thanks.
Chris W: You're very welcome John, I'll be right back with some information for you.
Chris W: Hi John, are you still with me?
John Grater: still here
Chris W: Great! I've created the escalation for you, and do apologize for not being able to resolve this issue with you at my level.
Chris W: You will be contacted via email by dotmac_tech_esc@apple.com in 24-48 hours. Please make sure that this email is on your accepted recipients list.
John Grater: OK. Thanks.
Chris W: You're welcome John. If you would like you can get a copy of this transcript sent to your email as well, so you have confirmation that they are supposed to contact you.
John Grater: That would be good - thanks.
Chris W: No problem, do you see the blue "Close" button in the top left? When you click on that you will be given an opportunity to have this transcript sent to you.
John Grater: Ok. Have a good day.
Chris W: You too John. I hope you have a great day, and would like to thank you for chatting with us. We value your feedback. Please click the blue "Close" button at top left to answer a few questions about your experience with us today."

Better write it all down

I've been trying to get the Back To My Mac facility of MobileMe working so that I can connect between my family homes around the globe. You can see a summary of the technical setup on the left. Having been through all the troubleshooting information on the Apple site and Google I had no option but to try to go through the Apple support system. Here's a log of what happened.