On 30 April, in a carefully planned data-gathering exercise, ESA's Mars Express team used the passage of Phobos between Earth and the spacecraft to record precisely when a radio signal was blocked. The data may now lead to the best-ever determination of Phobos's orbit, knowledge that will help scientists better understand our planetary neighbour and its double-moon system.
You'll recall our earlier posting, when we described MEX team plans to take advantage of a rare alignment of Earth - Phobos - Mars Express to measure precisely when the bulk of Phobos would block radio communications between the spacecraft and ESA's 35m Cebreros station in Spain.
The prediction was that the 'occultation' would last around 12 seconds, from 21:21:22.9 to 21:21:34.1 UTC (spacecraft time). The predicted ground receive time was between 21:29:13.0 and 21:29:24.2 UTC (i.e. 7:50 mins later on Earth).
The activity took place as planned on Monday, 30 April.
Based on a reconstruction of the MEX orbit by ESA's Flight Dynamics team at ESOC, results indicate that the expected start time of the occultation by Phobos was 0.5 seconds earlier than that predicted. Similarly, the end time is 0.4 seconds earlier than predicted.
The plot below indicates the actual situation as measured very precisely from Cebreros station.
It shows the view from Earth of the 'expected' occultation of the MEX signal by Phobos. The large, grey-green circle corresponds to the moon's disk; it is shown (at this scale) as a circle having a radius of 13.4 km.
So why was the actual recorded occultation times a few seconds shorter than expected?
ESA's Trevor Morley, lead flight dynamics specialist for Mars Express, says there are two possible reasons:
The best fit ellipsoid for the shape of Phobos has semi-axes of 13.4 x 11.2 x 9.2 km. The dashed circle shown in the plot has a radius of 9.2 km. The occultation duration depended upon the length of the chord across the real face of Phobos between the entry and exit points. The actual profile of Phobos needs to be applied to calculate this length (which is why photos/observations of Phobos are so important - Ed).
A shorter than expected occultation duration would also occur if the present error in our prediction of Phobos's orbit is such that the moon's true position at the time was further towards the south.
Trevor says that the recorded timing data will now be shared with the MEX science team as well as with colleagues at NASA JPL; it is expected that they will be able to use these results to improve models of the Phobos orbit:
“The Phobos orbit model that we are presently using was generated by Bob Jacobson of JPL and is almost certainly the most accurate one available. We will provide Bob with all the signal data plus our determined orbit of MEX,” he added.
“Of course we can provide the data to any scientists working on this topic, including those who have been using directional data derived from Phobos images made by the MEX camera for the same purpose.”
Mars Express Spacecraft Operations Manager Michel Denis says:
It's part of the scientific method to make use of every opportunity to investigate our natural world.
Also, it's second nature and ‘pride of craft’ for the mission operators to figure out how to achieve more with the same resources– whether that’s a spacecraft or a ground station. In this case, we could also achieve ‘extra science’ through tight cooperation between several teams here at ESOC – flight dynamics, ground segment, flight controllers – and we were able to fine-tune, second-by-second, an already packed science and communication plan.
On Monday, 30 April, Mars Express mission controllers at ESOC will take advantage of a rare alignment of the spacecraft with Phobos and Earth to use radio signals to determine the Martian moon's orbit.
A (very) rough sketch of the alignment
If the team's careful planning and preparation pay off, the occultation of Mars Express by Phobos (as seen from Earth) will provide highly accurate data that can be used to determine Phobos's orbit with unprecedented accuracy, possibly improving it by a factor of 2.
Essentially, here's what's going to happen:
On 30 April, Phobos will, for a few seconds, pass between MEX and Earth, blocking the direct line-of-sight path for radio signals transmitted from MEX to ESA's 35m deep-space station at Cebreros, Spain.
In a pointing specially extended for this event, Mars Express will already be turned toward Earth —
a mere pale blue dot some 140 mn km away
—
and will already be transmitting a steady signal to Cebreros.
Special radiometric recording equipment installed at the Cebreros ESTRACK station will record — with sub-second precision —
the start/finish time of the break in the transmission.
The occultation is predicted to last just 12 seconds, from 21:21:22.9 to 21:21:34.1 UTC (spacecraft time). The predicted ground receive time is between 21:29:13.0 and 21:29:24.2 UTC (i.e. 7:50 mins later on Earth).
Once the occultation ends, MEX will perform a 'high-speed' slew to re-orient itself so that the HRSC camera and other instruments are pointing toward Mars to conduct a regular observation pass over the Red Planet starting at 21:22 UTC.
Results from Cebreros station will be passed to ESA's flight dynamics team and to our colleagues at NASA's JPL, who will use these times in calculations that, once confirmed, could yield the best-ever determination of the moon's orbit.
"The occultation tracking promises to provide a very nice boost in our understanding of Phobos, an enigmatic body if ever there was one, without affecting routine science," says Mars Express Spacecraft Operations Manager Michel Denis.
We'll keep in touch with the Mars Express mission team at ESOC throughout the weekend and Monday/Tuesday to post any updates; you can also follow news via Twitter (@esaoperations).
The animation below shows what will happen immediately after the occultation tracking, when Mars Express re-orients itself for the Mars observation pass.
A quick link and extract from an update on Wednesday in ESA's Science directorate web pages; it provides a nice overview of the 'return to science' situation. References to our very own VMC camera activities highlighted - and note very nice comments on teamwork! Click link to read the full report.
While full science operations have now been resumed, a number of tasks remain to be completed. Most important among these is the implementation of an OBCP scheduler. This will enable the spacecraft to operate autonomously for up to a week, compared to the few days that are possible with the current FAST system. Work is also in hand to resume operation of the Visual Monitoring Camera (VMC – the 'Mars webcam').
Enormous team effort
Completely redesigning the way in which Mars Express is controlled has involved an enormous amount of work for the mission control team at the European Space Operations Centre (ESOC), assisted by their counterparts at the European Space Astronomy Centre (ESAC), PI-teams, other ESA experts and partners in industry. Everyone involved with the mission is extremely grateful for their hard work.
Although the 'Express' in Mars Express highlights that the mission was developed in a short time and with a relatively modest budget, the ability to resume full operations after a very serious failure shows that the resulting design is both robust and flexible.
Mars Express has now been restored to full operational capability and its potential mission lifetime remains unchanged
Last month, we spoke with several of the Mars Express team here at ESOC about their almost completed activities to restore, reconfigure and return Mars Express to service.
An interview with Mars Express Spacecraft Operations Engineer Daniel Lakey
Spacecraft Operations Engineer Daniel Lakey sitting beside the SSMM
A black box, edge-length 30 cm, is at the centre of the recent trouble with Mars Express.
Daniel Lakey, an engineer working on the mission at ESOC in Darmstadt, looks down at the engineering model of said black box sitting on his desk and recalls the seemingly endless night shifts he has had to pull because of its twin mounted on Mars Express, orbiting the Red Planet many millions of kilometres away.
In mid-August 2011, Mars Express unexpectedly placed itself into safe mode – think blue screen of death and reboot on a PC – because something went wrong either with the Solid-State Mass Memory (SSMM) housed inside this black box or with the on-board channels it uses to pass data to the spacecraft’s data management system (DMS) computer.
To extend the PC analogy, imagine that the memory chips in your computer, the RAM chips, or the memory controllers that tell them what to do, suffered a fault. The memory might continue functioning, apparently normal, but whenever an electronic signal tried to access the faulty unit, the operation would fail and the system would crash. That's what happened with Mars Express.
Holiday phone call at 3 AM
"I was on holidays in England when I got the call at three o'clock in the morning. Since I'm assigned as the mission's software coordinator, the problem fell in my area of responsibility," says Lakey.
Switching into safe mode means that the spacecraft automatically turns its solar panels to the Sun for maximum energy and its antenna to Earth for good communication – ostensibly very helpful in any untoward situation – but this process uses a significant amount of vital fuel. Every unnecessary safe mode reduces the life of this hugely valuable mission, and in safe mode, normal gathering of scientific data stops.
After an initial investigation, it was found that the safe modes were being triggered by the DMS computer whenever a batch of commands transferred from the SSMM was interrupted.
The problem: Big command batches were being interrupted,
triggering a fuel-gobbling safe mode
The SSMM is a large-capacity device, and it stores large numbers of commands sent by mission controllers and the instrument scientists, as well as raw data gathered by the instruments (prior to their being radioed back to Earth).
The SSMM then delivers a constant ‘stream’ of commands to the DMS computer one at a time; when the stream was interrupted – either due to a fault in the SSMM or due to some unknown problem with the on-board communication channels – the DMS detects the problem and auto-commands the spacecraft to switch to safe mode.
Taking action - but problems persist
At first, the flight control team executed the standard recovery procedures and restarted observations, hoping that Mars Express would function normally again.
But, frustratingly, safe modes happened two more times in the next few weeks, even though the engineers had tried switching on-board systems to use back-up communication channels (there is only one SSMM), among many other normal fixes. Nothing in the routine procedures, it seemed, could prevent the frustrating safe modes from occurring.
"We had to find a solution," says Lakey, "otherwise the mission would have soon been over."
By late August, the team had already gone through many night shifts trying to coax the recalcitrant spacecraft into some sort of stable configuration, with little luck.
"But then one day, an idea came to mind – while I was standing under the shower," says Lakey, with a laugh. "It occurred to me that, since something was happening to interrupt the flow of commands, triggering the safe mode, the solution might lie in by-passing the checks between the SSSM and the DMS computer, and finding a safe way to ignore problems with the link between the two."
With a little checking, Lakey was able to determine that the problem was, in fact, an issue of 'transient communication problems' between the SSMM and the computer. "When the main computer sees this interruption, it interprets it as a serious problem and stops executing its 'To-Do' list of commands – because it doesn't know whether the list is complete," says Lakey.
Fortunately, there's another, back-up, memory inside the DMS computer that could store the command stack, but it's much, much smaller than the SSMM, holding only 117 commands vs. over 3000.
So the engineers set about reconfiguring the spacecraft's systems to transfer commands from the SSMM to the onboard computer's memory in a different way. Rather than a constant stream of commands, one at a time, the commands would be transferred as a discrete block of commands relating to one complete spacecraft activity, just before that activity started.
"I thought we could use a trick, by packing the commands into smaller stacks and telling the on-board software to act only when it received a complete package. This 'all-or-nothing' scheme means we're no longer affected by the SSMM problems, but now we have more limits on what we can schedule in one go – but that's been proven to be acceptable."
But would they buy it?
As soon as he could, Lakey presented the idea to his colleagues.
"Perhaps predictably, they reacted with an operations engineer's traditional caution and scepticism. The first answers were, 'No, no, that won't work, No way...' But, after a lot of discussion, they slowly came around to 'Oh wait... maybe we should look at this... it could work'," Lakey tells.
The solution: make command batches small
With a clear consensus and the approval of Mars Express Spacecraft Operations Manager Michel Denis, the team set to work designing operations procedures that could be implemented using reduced command stacks, working first on just a certain set of basic on-board activities. This was a huge challenge.
As designed, Mars Express normally makes use of thousands of commands; for example, it takes up to 50 separate commands to simply take a single photo of Mars using the HRSC camera. Using the new, reduced command stacks would prove worthless if engineers couldn't actually do anything with the reduced command stacks.
Thus, making the solution work entails a massive amount of reprogramming to drastically reduce the number of commands needed to do anything on board. This work is what has kept the mission operations team on extended hours since November 2011.
Smiles all around
But, to everyone's delight, the solution is working and the team is substantially finished the work of converting thousands of commands to on-board procedures to be used much more efficiently than its designers had ever envisaged.
"We are confident that all the Mars Express instruments and systems can be commanded using the reduced command stacks," says Lakey.
"Now, we only need a few commands to capture an image and we can switch on and operate all the instruments at one time," he explains.
"We can proudly say that Mars Express is working properly again – and, with luck – the fuel left could last for another ten years."
The team examining the Engineering Model SSMM (solid state mass memory) during a Flight Control Team weekly 'FAST concept review' meeting. Left to Right are: Dan Lakey (DMS engineer), Erhard Rabenau (Mission Planning), Michel Denis (SOM), James Godfrey (Deputy-SOM), Olivier Reboud (AOCS/TTC Engineer)
A complete 'phase 1' set of instrument OBCPs (on-board control procedures) was uplinked successfully to Mars Express on 15 January, and following extensive ground validation - both on the software simulator and in the OBCP test & development environment, these have been flight tested on the spacecraft one-by-one over the last two weeks.
As of this week, all instruments can now be returned to routine operations. In addition, the radio science experiment (which has no dedicated instrument on board, and requires no OBCPs) will restart in two weeks' time. The OBCPs replace longer sequences of telecommands with macro 'on-board control procedures' - started with a single command - and allow several instruments to be operated simultaneously within the limited capacity (117 telecommands) of the 'Short-MTL' we now use for the on-board timeline execution.
A nice New Year's post starting the year with a question (and answer!) on the fabulous high resolution stereo camera (HRSC) image of Orcus Patera, first published by ESA (here) in 2010, and reposted as the Image of the Day on 31 December 2011 at the Mars Travel blog. Best wishes to everyone for a healthy and successful 2012. Details below! -- Daniel
First, here's the image (click image to access the original post in ESA web):
Orcus Patera is an enigmatic elliptical depression near Mars’s equator, in the eastern hemisphere of the planet. Located between the volcanoes of Elysium Mons and Olympus Mons, its formation remains a mystery. Often overlooked, this well-defined depression extends approximately 380 km by 140 km in a NNE–SSW direction. It has a rim that rises up to 1800 m above the surrounding plains, while the floor of the depression lies 400–600 m below the surroundings.
Earlier today, Mars Travel blogger David Geaney sent in a question:
If you look at one of the central craters in Orcus Patera using the HRSC image, you will see there is something blue in the crater. What is the blue material? Thank you for your time and patience.
We passed the query to Mars Express project scientist Olivier Witasse, who wrote:
Hi David,
The blue is not a lake....
It's dust, which emits in the blue part of the spectrum. In the data processing, the blue part is somewhat enhanced, which sometimes creates a wrong impression.
"The Mars Advanced Radar for Subsurface and Ionosphere Sounding (MARSIS) instrument on board Mars Express has recently completed a subsurface sounding campaign over the planet's North Pole. The campaign was interrupted by the suspension of science observations several times between August and October due to safe modes and to anomalies in the operation of the spacecraft's Solid-State Mass Memory (SSMM) system. As MARSIS best observes in the dark, which for the North Pole only occurs every few years, it was among the first instruments to resume observations once a partial work-around for the problems had been implemented."
In his comment below, the 'FAST Method' that Michel refers to is the operations team's newly developed way of uploading commands to Mars Express, which avoids using the problematic Solid State Mass Memory (SSMM) for critical commanding.
The 'File-based Activities on Short Timeline' method essentially means that commands are grouped in very short self-contained files that can be loaded safely, in advance of execution, from the SSMM into an alternative memory unit (that is reliable but not as capacious as the SSMM).
Michel writes:
The FAST method - loading short command files upon need into the short onboard mission timeline - was put into use at the end of October 2011 with the (excellent) result that we could save what was remaining of the North Pole observation campaign by the MARSIS radar.
The net loss in data collection was mitigated by using the existing MARSIS command sequences as soon as possible. Meanwhile, as for the other instruments, new MARSIS on-board control procedures (OBCPs) are under development and will allow operation with fewer commands, therefore enabling the operation of several science instruments in parallel.
My main point? We did our job: contrary to widespread received wisdom, the spacecraft operators' role is not to simply watch over (supposedly) boring routine operations during the many long years of a mission - nor simply saving a spacecraft that experiences problems. In fact, we are relied upon to deliver safely as much of the expected (precious) scientific data as possible within the resources available - despite adversity. And that's what we're doing!
This in this morning from Jonathan Schulster on the Mars Express operations team:
The ASPERA (Energetic Neutral Atoms Analyser) instrument high voltage (kV) lines and equipment were successfully switched on today, a few minutes ago at Mars (~09:57 CET), using the new on-board control procedures (OBCP).
These will run for one hour until 10:40CET today and the ASPERA science team will examine the recorded science data before giving the 'go-ahead' for full operations of ASPERA using only OBCP's starting 9 Jan uary 2012.
Cheers!
- Jonathan Schulster
Mars Express Mission Planning & Flight Control Team
Looks like another instrument is set to return to action! -- Daniel
Today's update comes courtesy of Jonathan Schulster, an engineer working on the Mars Express Mission Planning & Flight Control Team. Jonathan's in the Mars Express (MEX) Dedicated Control Room at ESOC this morning, where the first commands to switch on the ASPERA (Energetic Neutral Atoms Analyser) device were sent a few minutes ago. (ASPERA is studying the interaction between the solar wind and the Martian atmosphere.) -- Daniel
Jonathan writes:
To allow the mission to restart operations of all instruments, we needed to write 'macro' on-board control procedures (OBCPs) to replace long sequences of telecommands with single 'start macro' telecommands that would fit into the restrictive memory space provided by the short mission time-line (Basically, all commands now have to be a lot shorter to be stored on board now that the solid state mass memory no longer functions properly - Ed.).
The first OBCPs to switch ASPERA on and off, and its 'high voltage' lines up and down - along with on/off for PFS fourier spectrometer and the start/stop pre-heating for the OMEGA infrared (IR) spectrometer 'scanner' - were uplinked to Mars Express on Monday, 5 December. These must now be flight tested on the spacecraft.
The OMEGA pre-heating test took place on 7 December during a pass over ESA's 35m ground station at Cebreros, Spain, between 08:00 08:30 GMT (09:00-09:30 CET).
Today, Friday, 9 December, also between 08:00-08:30GMT, the ASPERA instrument switch on/off OBCPs were successfully flight tested. :-)
Pierre Choukroun (Standing, Left), Erhard Rabenau (standing, right) and Jonathan Schulster, sitting, in MEX DCR this morning.
Next week we plan to perform a full test of ASPERA with high voltage up/down after confirmation from the science team that the on/off worked fine.
Mission controllers are making excellent progress in returning Mars Express to routine service. Some science activities have already resumed after being temporarily suspended last month following a series of faults related to the onboard data storage system.
Having arrived around Mars in December 2003 for an initial two-year mission, the spacecraft is now in its eighth year of science operations. It has returned some of the most stunning images and valuable science data ever obtained from the Red Planet.
In mid-October 2011, anomalies in the operation of its Solid-State Mass Memory (SSMM) system caused science observations to be temporarily halted.
Anomalies in the operation of the solid-state mass memory system on board Mars Express have caused science observations to be temporarily halted. A technical work-around is being investigated that will enable the resumption of a number of observations and should evolve into a long-term solution.
In mid-August, Mars Express autonomously entered safe mode, an operational mode designed to safeguard both the spacecraft itself and its instrument payload in the event of faults or errors.
The cause of entering the safe mode was a complex combination of events relating to reading from and writing to memory modules in the Solid-State Mass Memory (SSMM) system. This is used to store data acquired by the instruments and housekeeping data from the spacecraft's subsystems, prior to its transmission to Earth, and is also used to store commands for the spacecraft that have been received from the ground stations, while awaiting execution.
Earlier this month, ESA’s Mars Express performed a special manoeuvre to observe an unusual alignment of Jupiter and the martian moon Phobos. The impressive images have now been processed into a movie of this rare event.
At the moment when Mars Express, Phobos, and Jupiter aligned on 1 June 2011, there was a distance of 11 389 km between the spacecraft and Phobos, and a further 529 million km to Jupiter.
A nice wrap-up on Mars Express achievements today!
Mars Express Spacecraft Operations Manager Michel Denis sent in anote pointing out that, according to ScienceWatch.com, in the 'Top 10 papers on planetary science of the last 10 years', the most-cited papers (5 out of 10) are from Mars Express data and scientists (ranking 1, 2, 4, 6 and 9). In the same list, quite a number of papers are Cassini- and Huygens-related. Full details via: http://www.sciencewatch.com.
As Michel mentions, this is a nice continuation of the 'Top 10 Science Insights of the Decade' in the magazine Science (December 2010) where, with all sciences together, 'Wate on Mars' appears as one of the ten (with Mars Express mentioned by name!). -- Daniel
A few days ago, Dan Brennen, of Elk Grove, California, posted a very nice animation in the UnmannedSpaceflight.com forums and we are delighted to share this below. Dan sent us a description:
The animation was created using the sequence of 5 images located here which were scaled down to 1/5 size. A simple morphing program was utilized to create the intermediate images which allow for the appearance of motion. Approximately 30 control points were identified to tell the program where the limb had changed shape and key features on Phobos had moved between images. Each transition from one image to the next contained 200 steps, which was later compressed to 8 frames per second to create the final image file.
I'm sure you can guess that the quality would improved with more control points and more precise placement of those control points. But it's quite time consuming, especially with five images. So I did this with less time and effort in order to quickly create an interesting feature for our forum in just an hour or two while I had my Saturday morning coffee.
Thanks, Dan, for some great work! -- Daniel
Original animation via UnMannedSpaceFlight.com Original HRSC images: Copyright ESA/DLR/FU Berlin (G. Neukum). Animation credit: Dan Brennan, Elk Grove, California
Emily Lakdawalla over at the Planetary Society Blog has posted a very detailed explanation of how the Mars Express High-Resolution Stereo Camera (HRSC) works, with some very informative examples and descriptions from this month's Phobos flyby. Emily writes:
But what I think is best about this particular set of images is that they serve as really great illustrations of how HRSC works. It's an unusual instrument: a pushbroom camera that acquires simultaneous stereo and color data with a higher-resolution framing camera bolted to it. If you don't understand what that preceding sentence means, bear with me; I'll explain.
Home
If the team's careful planning and preparation pay off, the occultation of Mars Express by Phobos (as seen from Earth) will provide highly accurate data that can be used to determine Phobos's orbit with unprecedented accuracy, possibly improving it by a factor of 2.
"Perhaps predictably, they reacted with an operations engineer's traditional caution and scepticism. The first answers were, 'No, no, that won't work, No way...' But, after a lot of discussion, they slowly came around to 'Oh wait... maybe we should look at this... it could work'," Lakey tells.
To allow the mission to restart operations of all instruments, we needed to write 'macro' on-board control procedures (OBCPs) to replace long sequences of telecommands with single 'start macro' telecommands that would fit into the restrictive memory space provided by the short mission time-line (Basically, all commands now have to be a lot shorter to be stored on board now that the solid state mass memory no longer functions properly - Ed.).
