Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

classic Classic list List threaded Threaded
15 messages Options
Clay Fenlason Clay Fenlason
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

I'm very happy to see this, Anne-Sophie, thank you. I know it's early
days, and there will be any number of ways people will want to improve
this practice as time goes on, but just setting the precedent seems to
me a significant step, and not only for the released product. We don't
have many UX professionals on staff (I think most Sakai institutions
are in the same boat), and if the Sakai community helps us learn how
to do this sort of work well it will be yet another advantage for our
participation.

~Clay

On Wed, Aug 17, 2011 at 4:28 PM, Anne-Sophie De Baets
<[hidden email]> wrote:

> Hi everyone,
>
> The Sakai OAE project team strongly believes that regular
> usability testing of our product across various institutions is
> essential in identifying problems, missing areas and improving
> overall usability. It is also a great way for the institutions to
> provide feedback and start exposing some of their staff and
> students to Sakai OAE.
>
> Therefore, Usability Test packages will be sent out from time
> to time, aiming to investigate specific areas of Sakai OAE. These
> packages will contain everything necessary for each institution
> to conduct the usability test themselves. These materials include
> a manual with tips and hints for people that have never conducted
> usability testing, the actual usability test guide and all other materials
> necessary. Each institution willing to participate will also receive
> its own (temporary) OAE URL that can be used for testing. All user
> accounts, data and materials necessary to run the usability test
> will be available on this.
>
> I'm proud to announce that the first Sakai OAE Usability
> Testing package is now available at [1]. Its goal is to
> investigate various management activities around content
> and courses/groups.
>
> We would like to encourage all institutions to participate in this.
> Please don't hesitate to get in touch with us if you would have
> any feedback or follow-up questions.
>
> [1] https://confluence.sakaiproject.org/display/3AK/Usability+Testing
>
>
> Have fun,
> Anne-Sophie De Baets
>
>
> _______________________________________________
> sakai-dev mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
>
_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Bruce D'Arcus-3 Bruce D'Arcus-3
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

On Thu, Aug 18, 2011 at 8:59 AM, Clay Fenlason
<[hidden email]> wrote:

> I'm very happy to see this, Anne-Sophie, thank you. I know it's early
> days, and there will be any number of ways people will want to improve
> this practice as time goes on, but just setting the precedent seems to
> me a significant step, and not only for the released product. We don't
> have many UX professionals on staff (I think most Sakai institutions
> are in the same boat), and if the Sakai community helps us learn how
> to do this sort of work well it will be yet another advantage for our
> participation.

Yes, absolutely!

Bruce
_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Allison Bloodworth Allison Bloodworth
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Clay Fenlason
Hi Anne-Sophie,

Thanks much for all the incredible work you've put into this! We are looking forward to starting testing soon.

Rachel & I are going over the protocol in detail now and it would be very helpful to actually run through it on a test server. However, we don't want to start Berkeley's countdown clock on having our server for 3 weeks quite yet--is there a server that is set up now that we could use to run through the test?

In our current context (as we have no courses in Sakai OAE) it would actually make more sense to test this on a group, e.g. a "Calculus Research Project group." This would likely also allow us to obtain information about how users in groups function. Do you think it would be problematic to make that change? 

Additionally, there are a few other things we'd like to add to the test to investigate the usability of other things that are important to us at Berkeley (e.g. categories, profile). We are hoping information from some additional tasks we add would still be helpful to the managed project?

Thanks again!
Allison

On Aug 18, 2011, at 5:59 AM, Clay Fenlason wrote:

I'm very happy to see this, Anne-Sophie, thank you. I know it's early
days, and there will be any number of ways people will want to improve
this practice as time goes on, but just setting the precedent seems to
me a significant step, and not only for the released product. We don't
have many UX professionals on staff (I think most Sakai institutions
are in the same boat), and if the Sakai community helps us learn how
to do this sort of work well it will be yet another advantage for our
participation.

~Clay

On Wed, Aug 17, 2011 at 4:28 PM, Anne-Sophie De Baets
<[hidden email]> wrote:
Hi everyone,

The Sakai OAE project team strongly believes that regular
usability testing of our product across various institutions is
essential in identifying problems, missing areas and improving
overall usability. It is also a great way for the institutions to
provide feedback and start exposing some of their staff and
students to Sakai OAE.

Therefore, Usability Test packages will be sent out from time
to time, aiming to investigate specific areas of Sakai OAE. These
packages will contain everything necessary for each institution
to conduct the usability test themselves. These materials include
a manual with tips and hints for people that have never conducted
usability testing, the actual usability test guide and all other materials
necessary. Each institution willing to participate will also receive
its own (temporary) OAE URL that can be used for testing. All user
accounts, data and materials necessary to run the usability test
will be available on this.

I'm proud to announce that the first Sakai OAE Usability
Testing package is now available at [1]. Its goal is to
investigate various management activities around content
and courses/groups.

We would like to encourage all institutions to participate in this.
Please don't hesitate to get in touch with us if you would have
any feedback or follow-up questions.

[1] https://confluence.sakaiproject.org/display/3AK/Usability+Testing


Have fun,
Anne-Sophie De Baets


_______________________________________________
sakai-dev mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"

_______________________________________________
sakai-dev mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"

Allison Bloodworth
Senior User Experience Designer
Educational Technology Services
University of California, Berkeley
510-289-6627








_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Bruce D'Arcus-3 Bruce D'Arcus-3
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Bruce D'Arcus-3
Really nicely done.

Question: given that the details of how you put these together the
test scenarios will be pretty critical in determining the quality of
results you get, might it be desirable to post them for feedback on
Confluence before you put together each package?

Example: as I've mentioned a few times on list, I have a suspicion
there are some usability problems around adding new content and
sharing it (that it's more complicated and time consuming that it
should be). But in a quick look, the first pack seems to only focus on
sharing content already in the system (though the language of the
description is a little vague to me; "sharing" is a generic term). If
that's the case, then it will miss problems I would anticipate would
be exposed more easily if the content needed to be added to the system
from somewhere else (Word files, Flick images, New York Times
articles, etc.).

Also, as Allison points out, the scenario assumes sharing by
instructors in a class, but is that restriction really necessary?

Bruce
_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Clay Fenlason Clay Fenlason
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Allison Bloodworth
Nico and I are working on getting some Amazon instances up for
usability testing, including pre-populated data IIUC, and I hope we'll
get that out to you soon.

A key part of the goal in my mind is to support evidence-based design
decisions, as opposed to merely trying to compromise between several
institutional opinions. To that end I think it would be more important
to start by finding a common set of tests we can all run. For similar
reasons, I'm leery of formulations like "the usability of things that
are important to Berkeley." I don't want to read too much into that
phrase, but if we accept it uncritically we could find ourselves
sliding back into a process of negotiating opinions and a
'contrib/core' mentality that wouldn't help us achieve overall
coherence. It would at least be premature, as all our design issues
are fairly general at this stage.

Do we have the beginning of a group of people who could work to evolve
and refine the test suite?

~Clay

On Fri, Aug 19, 2011 at 3:39 PM, Allison Bloodworth
<[hidden email]> wrote:

> Hi Anne-Sophie,
> Thanks much for all the incredible work you've put into this! We are looking
> forward to starting testing soon.
>
> Rachel & I are going over the protocol in detail now and it would be very
> helpful to actually run through it on a test server. However, we don't want
> to start Berkeley's countdown clock on having our server for 3 weeks quite
> yet--is there a server that is set up now that we could use to run through
> the test?
> In our current context (as we have no courses in Sakai OAE) it would
> actually make more sense to test this on a group, e.g. a "Calculus Research
> Project group." This would likely also allow us to obtain information about
> how users in groups function. Do you think it would be problematic to make
> that change?
> Additionally, there are a few other things we'd like to add to the test to
> investigate the usability of other things that are important to us at
> Berkeley (e.g. categories, profile). We are hoping information from some
> additional tasks we add would still be helpful to the managed project?
> Thanks again!
> Allison
> On Aug 18, 2011, at 5:59 AM, Clay Fenlason wrote:
>
> I'm very happy to see this, Anne-Sophie, thank you. I know it's early
> days, and there will be any number of ways people will want to improve
> this practice as time goes on, but just setting the precedent seems to
> me a significant step, and not only for the released product. We don't
> have many UX professionals on staff (I think most Sakai institutions
> are in the same boat), and if the Sakai community helps us learn how
> to do this sort of work well it will be yet another advantage for our
> participation.
>
> ~Clay
>
> On Wed, Aug 17, 2011 at 4:28 PM, Anne-Sophie De Baets
> <[hidden email]> wrote:
>
> Hi everyone,
>
> The Sakai OAE project team strongly believes that regular
>
> usability testing of our product across various institutions is
>
> essential in identifying problems, missing areas and improving
>
> overall usability. It is also a great way for the institutions to
>
> provide feedback and start exposing some of their staff and
>
> students to Sakai OAE.
>
> Therefore, Usability Test packages will be sent out from time
>
> to time, aiming to investigate specific areas of Sakai OAE. These
>
> packages will contain everything necessary for each institution
>
> to conduct the usability test themselves. These materials include
>
> a manual with tips and hints for people that have never conducted
>
> usability testing, the actual usability test guide and all other materials
>
> necessary. Each institution willing to participate will also receive
>
> its own (temporary) OAE URL that can be used for testing. All user
>
> accounts, data and materials necessary to run the usability test
>
> will be available on this.
>
> I'm proud to announce that the first Sakai OAE Usability
>
> Testing package is now available at [1]. Its goal is to
>
> investigate various management activities around content
>
> and courses/groups.
>
> We would like to encourage all institutions to participate in this.
>
> Please don't hesitate to get in touch with us if you would have
>
> any feedback or follow-up questions.
>
> [1] https://confluence.sakaiproject.org/display/3AK/Usability+Testing
>
>
> Have fun,
>
> Anne-Sophie De Baets
>
>
> _______________________________________________
>
> sakai-dev mailing list
>
> [hidden email]
>
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to [hidden email]
> with a subject of "unsubscribe"
>
> _______________________________________________
> sakai-dev mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to [hidden email]
> with a subject of "unsubscribe"
>
> Allison Bloodworth
> Senior User Experience Designer
> Educational Technology Services
> University of California, Berkeley
> [hidden email]
> 510-289-6627
>
>
>
>
>
>
>
> _______________________________________________
> sakai-ui-dev mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev
>
>
_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Anne-Sophie De Baets Anne-Sophie De Baets
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Allison Bloodworth
Hi Allison,

Thanks a lot for your feedback. Some comments inline:

> Rachel & I are going over the protocol in detail now and it would be very
> helpful to actually run through it on a test server. However, we don't
> want to start Berkeley's countdown clock on having our server for 3 weeks
> quite yet--is there a server that is set up now that we could use to run
> through the test?

That makes sense. You can experiment with the 3akai.sakaiproject.org
server, using the same accounts and data as described in the manual,
which have been created on there as well. However, the server is not
running the latest code, so bugs might still exist. We'll reset the
server to its initial state from time to time, meaning that you will
lose all of the changes you make to the test accounts and data.

> In our current context (as we have no courses in Sakai OAE) it would
> actually make more sense to test this on a group, e.g. a "Calculus
> Research Project group." This would likely also allow us to obtain
> information about how users in groups function. Do you think it would be
> problematic to make that change?

In general, it would be good if we could keep the testing scenario
as close to the original test as possible, just to make it easier
to analyse results coming from different institutions. However, in
this case, I think the change you'd like to make should be fine. You'll
still be testing the same functionality and interface elements, as
you're basically just changing the name and type.

It's probably worth keeping in mind that the server will already have
the Maths course pre-created for you. This means that you'll have
to create the Mathematics Research Project group yourself and modify
the content of the messages present in the system. It is important
to keep the content of the group and messages as close as possible
to the originals.

> Additionally, there are a few other things we'd like to add to the test to
> investigate the usability of other things that are important to us at
> Berkeley (e.g. categories, profile). We are hoping information from some
> additional tasks we add would still be helpful to the managed project?

I think it's great that you're keen to test other things as well, but
for this first package I would encourage institutions to run the test
as closely as possible to the original without changing or expanding
the scope and goals. We're still trying to figure out how we can
best analyse and summarise results coming from different institutions,
so for now it would be more helpful if all participating institutions
conducted the same focused test.

Another reason why it might not be the best time to test categories and
profiles is that both of these are currently undergoing design changes.
It feels more useful to create a usability package that tests these
changes, rather than the current solution which will disappear soon.

However, it's really valuable to find out what you would like to test
as well. The goal is to send out small focused usability test packages
at regular times, and that information could help us in deciding what
to test next. Tomorrow, I will create a Confluence page where everyone
can list what they would like to test in the next usability package. I'll
send out a note once that page is up.

Best wishes,
Anne-Sophie

_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Anne-Sophie De Baets Anne-Sophie De Baets
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Bruce D'Arcus-3
Thanks Bruce.

> Question: given that the details of how you put these together the
> test scenarios will be pretty critical in determining the quality of
> results you get, might it be desirable to post them for feedback on
> Confluence before you put together each package?

That's absolutely true. Tomorrow, I will create a Confluence page where
everyone can list what they would like to see tested in the next usability
package. I'll send out a note once that page is up.

We can also add a draft of each usability testing package to Confluence
before we actually send it out.

Best wishes,
Anne-Sophie



_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Allison Bloodworth Allison Bloodworth
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Clay Fenlason
Hi folks,

I want to reassure you that "the usability of things that are important at Berkeley" wasn't meant to be a statement that is in any way divisive or indicate that we are trying to push the product in a direction that inhibits overall coherence. It was just an honest statement, based on the fact that I don't believe certain things that are important to us right now may be as important to other institutions at the moment.

As a UX practitioner, when I plan a usability test I definitely define tasks that help me understand key issues I'm working on. For example, categories are quite important to us as we feel like this is going to be an important navigational method for our users. [1] However, I am not even sure if any other institutions are using categories so I wouldn't necessarily expect them to be interested in spending precious time with users testing it.

We thought about adding tasks because seemed like the tasks outlined wouldn't take very long (we often have hour-long usability test sessions). If we started the sessions with the tasks in the protocol, I'm sure it wouldn't change the results to add a few more at the end. As you can see from the instructions it is quite a bit of work & coordination to get users together for testing sessions, so adding tasks would help us limit our coordination overhead and make better use of our time with users. We were hoping, however, that at Berkeley we could add tasks that would both inform the managed project and help answer questions we have locally.

As a final note, I wasn't aware that there were changes coming soon in categories & profile or I likely would not have suggested testing them (unless of course we could get input to the managed project before folks start work on these changes). I know we have the 90 day plan [2], but I'm not aware of a way to tell which work is coming next--is there a place to find that info? It may also be helpful to have a discussion with Nico of the places in the system that he and Sam view as "unfinished" and not worthy of doing any testing.

Cheers,
Allison

[1] Here is a screenshot, which you can also check out live at calcentral.berkeley.edu, of our categories in case folks are interested in how we've worked certain issues around not having enough hierarchy:




[2] https://confluence.sakaiproject.org/display/3AK/Sakai+OAE+v1.1+90-day+plan

On Aug 22, 2011, at 6:44 AM, Clay Fenlason wrote:

> Nico and I are working on getting some Amazon instances up for
> usability testing, including pre-populated data IIUC, and I hope we'll
> get that out to you soon.
>
> A key part of the goal in my mind is to support evidence-based design
> decisions, as opposed to merely trying to compromise between several
> institutional opinions. To that end I think it would be more important
> to start by finding a common set of tests we can all run. For similar
> reasons, I'm leery of formulations like "the usability of things that
> are important to Berkeley." I don't want to read too much into that
> phrase, but if we accept it uncritically we could find ourselves
> sliding back into a process of negotiating opinions and a
> 'contrib/core' mentality that wouldn't help us achieve overall
> coherence. It would at least be premature, as all our design issues
> are fairly general at this stage.
>
> Do we have the beginning of a group of people who could work to evolve
> and refine the test suite?
>
> ~Clay
>
> On Fri, Aug 19, 2011 at 3:39 PM, Allison Bloodworth
> <[hidden email]> wrote:
>> Hi Anne-Sophie,
>> Thanks much for all the incredible work you've put into this! We are looking
>> forward to starting testing soon.
>>
>> Rachel & I are going over the protocol in detail now and it would be very
>> helpful to actually run through it on a test server. However, we don't want
>> to start Berkeley's countdown clock on having our server for 3 weeks quite
>> yet--is there a server that is set up now that we could use to run through
>> the test?
>> In our current context (as we have no courses in Sakai OAE) it would
>> actually make more sense to test this on a group, e.g. a "Calculus Research
>> Project group." This would likely also allow us to obtain information about
>> how users in groups function. Do you think it would be problematic to make
>> that change?
>> Additionally, there are a few other things we'd like to add to the test to
>> investigate the usability of other things that are important to us at
>> Berkeley (e.g. categories, profile). We are hoping information from some
>> additional tasks we add would still be helpful to the managed project?
>> Thanks again!
>> Allison
>> On Aug 18, 2011, at 5:59 AM, Clay Fenlason wrote:
>>
>> I'm very happy to see this, Anne-Sophie, thank you. I know it's early
>> days, and there will be any number of ways people will want to improve
>> this practice as time goes on, but just setting the precedent seems to
>> me a significant step, and not only for the released product. We don't
>> have many UX professionals on staff (I think most Sakai institutions
>> are in the same boat), and if the Sakai community helps us learn how
>> to do this sort of work well it will be yet another advantage for our
>> participation.
>>
>> ~Clay
>>
>> On Wed, Aug 17, 2011 at 4:28 PM, Anne-Sophie De Baets
>> <[hidden email]> wrote:
>>
>> Hi everyone,
>>
>> The Sakai OAE project team strongly believes that regular
>>
>> usability testing of our product across various institutions is
>>
>> essential in identifying problems, missing areas and improving
>>
>> overall usability. It is also a great way for the institutions to
>>
>> provide feedback and start exposing some of their staff and
>>
>> students to Sakai OAE.
>>
>> Therefore, Usability Test packages will be sent out from time
>>
>> to time, aiming to investigate specific areas of Sakai OAE. These
>>
>> packages will contain everything necessary for each institution
>>
>> to conduct the usability test themselves. These materials include
>>
>> a manual with tips and hints for people that have never conducted
>>
>> usability testing, the actual usability test guide and all other materials
>>
>> necessary. Each institution willing to participate will also receive
>>
>> its own (temporary) OAE URL that can be used for testing. All user
>>
>> accounts, data and materials necessary to run the usability test
>>
>> will be available on this.
>>
>> I'm proud to announce that the first Sakai OAE Usability
>>
>> Testing package is now available at [1]. Its goal is to
>>
>> investigate various management activities around content
>>
>> and courses/groups.
>>
>> We would like to encourage all institutions to participate in this.
>>
>> Please don't hesitate to get in touch with us if you would have
>>
>> any feedback or follow-up questions.
>>
>> [1] https://confluence.sakaiproject.org/display/3AK/Usability+Testing
>>
>>
>> Have fun,
>>
>> Anne-Sophie De Baets
>>
>>
>> _______________________________________________
>>
>> sakai-dev mailing list
>>
>> [hidden email]
>>
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to [hidden email]
>> with a subject of "unsubscribe"
>>
>> _______________________________________________
>> sakai-dev mailing list
>> [hidden email]
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to [hidden email]
>> with a subject of "unsubscribe"
>>
>> Allison Bloodworth
>> Senior User Experience Designer
>> Educational Technology Services
>> University of California, Berkeley
>> [hidden email]
>> 510-289-6627
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> sakai-ui-dev mailing list
>> [hidden email]
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev
>>
>>
> _______________________________________________
> sakai-dev mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Allison Bloodworth
Senior User Experience Designer
Educational Technology Services
University of California, Berkeley
[hidden email]
510-289-6627








_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"

Screen shot 2011-08-22 at 4.34.33 PM.png (182K) Download Attachment
Allison Bloodworth Allison Bloodworth
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Anne-Sophie De Baets
Hi Anne-Sophie,

Thanks much for your responses and patience with us as Rachel, Daphne & I came up with this feedback after taking a closer look at the test together. :) Comments below...

On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:

> Hi Allison,
>
> Thanks a lot for your feedback. Some comments inline:
>
>> Rachel & I are going over the protocol in detail now and it would be very
>> helpful to actually run through it on a test server. However, we don't
>> want to start Berkeley's countdown clock on having our server for 3 weeks
>> quite yet--is there a server that is set up now that we could use to run
>> through the test?
>
> That makes sense. You can experiment with the 3akai.sakaiproject.org
> server, using the same accounts and data as described in the manual,
> which have been created on there as well. However, the server is not
> running the latest code, so bugs might still exist. We'll reset the
> server to its initial state from time to time, meaning that you will
> lose all of the changes you make to the test accounts and data.

Great, thanks! We took a quick look here and the server is a bit slow but this should help us get an idea before we get our own server.

>
>> In our current context (as we have no courses in Sakai OAE) it would
>> actually make more sense to test this on a group, e.g. a "Calculus
>> Research Project group." This would likely also allow us to obtain
>> information about how users in groups function. Do you think it would be
>> problematic to make that change?
>
> In general, it would be good if we could keep the testing scenario
> as close to the original test as possible, just to make it easier
> to analyse results coming from different institutions. However, in
> this case, I think the change you'd like to make should be fine. You'll
> still be testing the same functionality and interface elements, as
> you're basically just changing the name and type.
>
> It's probably worth keeping in mind that the server will already have
> the Maths course pre-created for you. This means that you'll have
> to create the Mathematics Research Project group yourself and modify
> the content of the messages present in the system. It is important
> to keep the content of the group and messages as close as possible
> to the originals.

We understand that there will be extra work involved if we make this change, and will will weigh this when deciding whether or not to do it. We definitely agree that the goals should be the same.

>
>> Additionally, there are a few other things we'd like to add to the test to
>> investigate the usability of other things that are important to us at
>> Berkeley (e.g. categories, profile). We are hoping information from some
>> additional tasks we add would still be helpful to the managed project?
>
> I think it's great that you're keen to test other things as well, but
> for this first package I would encourage institutions to run the test
> as closely as possible to the original without changing or expanding
> the scope and goals. We're still trying to figure out how we can
> best analyse and summarise results coming from different institutions,
> so for now it would be more helpful if all participating institutions
> conducted the same focused test.

We agree that analysis will be difficult. We had talked with Nico about the new OAE UX group (of which we hope you will be a part! :)) working on this. Are you envisioning the Sakai OAE UX group analyzing the results in some way together, or that you would take the lead on analysis and the rest of us would assist you in some fashion? Perhaps we can talk sometime soon about how we can help. We have been meeting (irregularly so far) Tuesdays at 9am PDT if that happens to work for you, or we'd be happy to set up a time anytime this week.

>
> Another reason why it might not be the best time to test categories and
> profiles is that both of these are currently undergoing design changes.
> It feels more useful to create a usability package that tests these
> changes, rather than the current solution which will disappear soon.

Great, that's exactly the sort of feedback we were looking for. Not testing these things makes sense if the changes aren't out by the time we do our tests. Any idea when these changes will be made?

>
> However, it's really valuable to find out what you would like to test
> as well. The goal is to send out small focused usability test packages
> at regular times, and that information could help us in deciding what
> to test next. Tomorrow, I will create a Confluence page where everyone
> can list what they would like to test in the next usability package. I'll
> send out a note once that page is up.

It would be great to discuss strategy for future usability testing sessions together--and we think it's a fantastic idea to create a confluence page to discuss what to test next!  Since finding & scheduling users takes us quite a bit of time (as we endeavor to get a realistic sample as opposed to doing 'friends & family (or colleague)' testing), it would be easier for us to have fewer testing sessions which cover more ground. I'd be interested to hear about how other institutions who plan to participate in the testing sessions feel.

Cheers,
Allison

>
> Best wishes,
> Anne-Sophie
>

Allison Bloodworth
Senior User Experience Designer
Educational Technology Services
University of California, Berkeley
[hidden email]
510-289-6627







_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Anne-Sophie De Baets Anne-Sophie De Baets
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Anne-Sophie De Baets
Hello,

I've now added a subpage [1] where institutions can make suggestions for
future test packages.

Anne-Sophie

[1] https://confluence.sakaiproject.org/display/3AK/Future+Usability+Packages


> Thanks Bruce.
>
>> Question: given that the details of how you put these together the
>> test scenarios will be pretty critical in determining the quality of
>> results you get, might it be desirable to post them for feedback on
>> Confluence before you put together each package?
>
> That's absolutely true. Tomorrow, I will create a Confluence page where
> everyone can list what they would like to see tested in the next usability
> package. I'll send out a note once that page is up.
>
> We can also add a draft of each usability testing package to Confluence
> before we actually send it out.
>
> Best wishes,
> Anne-Sophie
>
>
>


_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Allison Bloodworth Allison Bloodworth
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Allison Bloodworth
Hi folks,

At UC Berkeley we've completed 4 usability tests and have 6 more scheduled by the end of next Tuesday. We decided not to modify the protocol by changing it to a project site or adding additional tasks.

We have learned some things about the test environment --including a change that is needed to the permissions in the environment before starting the test, and some cleanup of the environment that is necessary between tests because the same "Elementary Calculus" course is being used by each tester.

Is anyone else doing usability testing? If so I'll write up what we've learned about these things in detail.

Cheers,
Allison

On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:

> Hi Anne-Sophie,
>
> Thanks much for your responses and patience with us as Rachel, Daphne & I came up with this feedback after taking a closer look at the test together. :) Comments below...
>
> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
>
>> Hi Allison,
>>
>> Thanks a lot for your feedback. Some comments inline:
>>
>>> Rachel & I are going over the protocol in detail now and it would be very
>>> helpful to actually run through it on a test server. However, we don't
>>> want to start Berkeley's countdown clock on having our server for 3 weeks
>>> quite yet--is there a server that is set up now that we could use to run
>>> through the test?
>>
>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>> server, using the same accounts and data as described in the manual,
>> which have been created on there as well. However, the server is not
>> running the latest code, so bugs might still exist. We'll reset the
>> server to its initial state from time to time, meaning that you will
>> lose all of the changes you make to the test accounts and data.
>
> Great, thanks! We took a quick look here and the server is a bit slow but this should help us get an idea before we get our own server.
>
>>
>>> In our current context (as we have no courses in Sakai OAE) it would
>>> actually make more sense to test this on a group, e.g. a "Calculus
>>> Research Project group." This would likely also allow us to obtain
>>> information about how users in groups function. Do you think it would be
>>> problematic to make that change?
>>
>> In general, it would be good if we could keep the testing scenario
>> as close to the original test as possible, just to make it easier
>> to analyse results coming from different institutions. However, in
>> this case, I think the change you'd like to make should be fine. You'll
>> still be testing the same functionality and interface elements, as
>> you're basically just changing the name and type.
>>
>> It's probably worth keeping in mind that the server will already have
>> the Maths course pre-created for you. This means that you'll have
>> to create the Mathematics Research Project group yourself and modify
>> the content of the messages present in the system. It is important
>> to keep the content of the group and messages as close as possible
>> to the originals.
>
> We understand that there will be extra work involved if we make this change, and will will weigh this when deciding whether or not to do it. We definitely agree that the goals should be the same.
>
>>
>>> Additionally, there are a few other things we'd like to add to the test to
>>> investigate the usability of other things that are important to us at
>>> Berkeley (e.g. categories, profile). We are hoping information from some
>>> additional tasks we add would still be helpful to the managed project?
>>
>> I think it's great that you're keen to test other things as well, but
>> for this first package I would encourage institutions to run the test
>> as closely as possible to the original without changing or expanding
>> the scope and goals. We're still trying to figure out how we can
>> best analyse and summarise results coming from different institutions,
>> so for now it would be more helpful if all participating institutions
>> conducted the same focused test.
>
> We agree that analysis will be difficult. We had talked with Nico about the new OAE UX group (of which we hope you will be a part! :)) working on this. Are you envisioning the Sakai OAE UX group analyzing the results in some way together, or that you would take the lead on analysis and the rest of us would assist you in some fashion? Perhaps we can talk sometime soon about how we can help. We have been meeting (irregularly so far) Tuesdays at 9am PDT if that happens to work for you, or we'd be happy to set up a time anytime this week.
>
>>
>> Another reason why it might not be the best time to test categories and
>> profiles is that both of these are currently undergoing design changes.
>> It feels more useful to create a usability package that tests these
>> changes, rather than the current solution which will disappear soon.
>
> Great, that's exactly the sort of feedback we were looking for. Not testing these things makes sense if the changes aren't out by the time we do our tests. Any idea when these changes will be made?
>
>>
>> However, it's really valuable to find out what you would like to test
>> as well. The goal is to send out small focused usability test packages
>> at regular times, and that information could help us in deciding what
>> to test next. Tomorrow, I will create a Confluence page where everyone
>> can list what they would like to test in the next usability package. I'll
>> send out a note once that page is up.
>
> It would be great to discuss strategy for future usability testing sessions together--and we think it's a fantastic idea to create a confluence page to discuss what to test next!  Since finding & scheduling users takes us quite a bit of time (as we endeavor to get a realistic sample as opposed to doing 'friends & family (or colleague)' testing), it would be easier for us to have fewer testing sessions which cover more ground. I'd be interested to hear about how other institutions who plan to participate in the testing sessions feel.
>
> Cheers,
> Allison
>
>>
>> Best wishes,
>> Anne-Sophie
>>
>
> Allison Bloodworth
> Senior User Experience Designer
> Educational Technology Services
> University of California, Berkeley
> [hidden email]
> 510-289-6627
>
>
>
>
>
>
>
> _______________________________________________
> sakai-ui-dev mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev

Allison Bloodworth
Senior User Experience Designer
Educational Technology Services
University of California, Berkeley
[hidden email]
510-289-6627







_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Keli Sato Amann Keli Sato Amann
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [oae-urg] [Building Sakai] Sakai OAE Usability Testing Package 1

We have not, but we'd love to hear about your results.

----- Original Message -----
From: "Allison Bloodworth" <[hidden email]>
To: "Sakai UI Development" <[hidden email]>
Cc: "Anne-Sophie De Baets" <[hidden email]>, "Sakai OAE User Reference Group" <[hidden email]>, "Sakai UX" <[hidden email]>, "Sakai Developers" <[hidden email]>
Sent: Friday, September 9, 2011 9:27:54 AM
Subject: Re: [oae-urg] [Building Sakai] Sakai OAE Usability Testing Package 1

Hi folks,

At UC Berkeley we've completed 4 usability tests and have 6 more scheduled by the end of next Tuesday. We decided not to modify the protocol by changing it to a project site or adding additional tasks.

We have learned some things about the test environment --including a change that is needed to the permissions in the environment before starting the test, and some cleanup of the environment that is necessary between tests because the same "Elementary Calculus" course is being used by each tester.

Is anyone else doing usability testing? If so I'll write up what we've learned about these things in detail.

Cheers,
Allison

On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:

> Hi Anne-Sophie,
>
> Thanks much for your responses and patience with us as Rachel, Daphne & I came up with this feedback after taking a closer look at the test together. :) Comments below...
>
> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
>
>> Hi Allison,
>>
>> Thanks a lot for your feedback. Some comments inline:
>>
>>> Rachel & I are going over the protocol in detail now and it would be very
>>> helpful to actually run through it on a test server. However, we don't
>>> want to start Berkeley's countdown clock on having our server for 3 weeks
>>> quite yet--is there a server that is set up now that we could use to run
>>> through the test?
>>
>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>> server, using the same accounts and data as described in the manual,
>> which have been created on there as well. However, the server is not
>> running the latest code, so bugs might still exist. We'll reset the
>> server to its initial state from time to time, meaning that you will
>> lose all of the changes you make to the test accounts and data.
>
> Great, thanks! We took a quick look here and the server is a bit slow but this should help us get an idea before we get our own server.
>
>>
>>> In our current context (as we have no courses in Sakai OAE) it would
>>> actually make more sense to test this on a group, e.g. a "Calculus
>>> Research Project group." This would likely also allow us to obtain
>>> information about how users in groups function. Do you think it would be
>>> problematic to make that change?
>>
>> In general, it would be good if we could keep the testing scenario
>> as close to the original test as possible, just to make it easier
>> to analyse results coming from different institutions. However, in
>> this case, I think the change you'd like to make should be fine. You'll
>> still be testing the same functionality and interface elements, as
>> you're basically just changing the name and type.
>>
>> It's probably worth keeping in mind that the server will already have
>> the Maths course pre-created for you. This means that you'll have
>> to create the Mathematics Research Project group yourself and modify
>> the content of the messages present in the system. It is important
>> to keep the content of the group and messages as close as possible
>> to the originals.
>
> We understand that there will be extra work involved if we make this change, and will will weigh this when deciding whether or not to do it. We definitely agree that the goals should be the same.
>
>>
>>> Additionally, there are a few other things we'd like to add to the test to
>>> investigate the usability of other things that are important to us at
>>> Berkeley (e.g. categories, profile). We are hoping information from some
>>> additional tasks we add would still be helpful to the managed project?
>>
>> I think it's great that you're keen to test other things as well, but
>> for this first package I would encourage institutions to run the test
>> as closely as possible to the original without changing or expanding
>> the scope and goals. We're still trying to figure out how we can
>> best analyse and summarise results coming from different institutions,
>> so for now it would be more helpful if all participating institutions
>> conducted the same focused test.
>
> We agree that analysis will be difficult. We had talked with Nico about the new OAE UX group (of which we hope you will be a part! :)) working on this. Are you envisioning the Sakai OAE UX group analyzing the results in some way together, or that you would take the lead on analysis and the rest of us would assist you in some fashion? Perhaps we can talk sometime soon about how we can help. We have been meeting (irregularly so far) Tuesdays at 9am PDT if that happens to work for you, or we'd be happy to set up a time anytime this week.
>
>>
>> Another reason why it might not be the best time to test categories and
>> profiles is that both of these are currently undergoing design changes.
>> It feels more useful to create a usability package that tests these
>> changes, rather than the current solution which will disappear soon.
>
> Great, that's exactly the sort of feedback we were looking for. Not testing these things makes sense if the changes aren't out by the time we do our tests. Any idea when these changes will be made?
>
>>
>> However, it's really valuable to find out what you would like to test
>> as well. The goal is to send out small focused usability test packages
>> at regular times, and that information could help us in deciding what
>> to test next. Tomorrow, I will create a Confluence page where everyone
>> can list what they would like to test in the next usability package. I'll
>> send out a note once that page is up.
>
> It would be great to discuss strategy for future usability testing sessions together--and we think it's a fantastic idea to create a confluence page to discuss what to test next!  Since finding & scheduling users takes us quite a bit of time (as we endeavor to get a realistic sample as opposed to doing 'friends & family (or colleague)' testing), it would be easier for us to have fewer testing sessions which cover more ground. I'd be interested to hear about how other institutions who plan to participate in the testing sessions feel.
>
> Cheers,
> Allison
>
>>
>> Best wishes,
>> Anne-Sophie
>>
>
> Allison Bloodworth
> Senior User Experience Designer
> Educational Technology Services
> University of California, Berkeley
> [hidden email]
> 510-289-6627
>
>
>
>
>
>
>
> _______________________________________________
> sakai-ui-dev mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev

Allison Bloodworth
Senior User Experience Designer
Educational Technology Services
University of California, Berkeley
[hidden email]
510-289-6627







_______________________________________________
oae-urg mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/oae-urg
_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Daphne Ogle Daphne Ogle
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [oae-urg] [Building Sakai] Sakai OAE Usability Testing Package 1

Hi Keli,

We'll be sharing our results back to Anne-Sophie in the form of the  
profiles she's asked along with our raw notes and video.  For privacy  
reasons, the raw notes and video should be kept private within the  
project team (do we have a way to post these things and restrict  
access?) but my assumption is Anne-Sophie will make all the profiles  
public.  We should have those available next week.

-Daphne

On Sep 9, 2011, at 10:16 AM, Keli Sato Amann wrote:

> We have not, but we'd love to hear about your results.
>
> ----- Original Message -----
> From: "Allison Bloodworth" <[hidden email]>
> To: "Sakai UI Development" <[hidden email]>
> Cc: "Anne-Sophie De Baets" <[hidden email]>, "Sakai OAE User  
> Reference Group" <[hidden email]>, "Sakai UX" <[hidden email]
> >, "Sakai Developers" <[hidden email]>
> Sent: Friday, September 9, 2011 9:27:54 AM
> Subject: Re: [oae-urg] [Building Sakai] Sakai OAE Usability Testing  
> Package 1
>
> Hi folks,
>
> At UC Berkeley we've completed 4 usability tests and have 6 more  
> scheduled by the end of next Tuesday. We decided not to modify the  
> protocol by changing it to a project site or adding additional tasks.
>
> We have learned some things about the test environment --including a  
> change that is needed to the permissions in the environment before  
> starting the test, and some cleanup of the environment that is  
> necessary between tests because the same "Elementary Calculus"  
> course is being used by each tester.
>
> Is anyone else doing usability testing? If so I'll write up what  
> we've learned about these things in detail.
>
> Cheers,
> Allison
>
> On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:
>
>> Hi Anne-Sophie,
>>
>> Thanks much for your responses and patience with us as Rachel,  
>> Daphne & I came up with this feedback after taking a closer look at  
>> the test together. :) Comments below...
>>
>> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
>>
>>> Hi Allison,
>>>
>>> Thanks a lot for your feedback. Some comments inline:
>>>
>>>> Rachel & I are going over the protocol in detail now and it would  
>>>> be very
>>>> helpful to actually run through it on a test server. However, we  
>>>> don't
>>>> want to start Berkeley's countdown clock on having our server for  
>>>> 3 weeks
>>>> quite yet--is there a server that is set up now that we could use  
>>>> to run
>>>> through the test?
>>>
>>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>>> server, using the same accounts and data as described in the manual,
>>> which have been created on there as well. However, the server is not
>>> running the latest code, so bugs might still exist. We'll reset the
>>> server to its initial state from time to time, meaning that you will
>>> lose all of the changes you make to the test accounts and data.
>>
>> Great, thanks! We took a quick look here and the server is a bit  
>> slow but this should help us get an idea before we get our own  
>> server.
>>
>>>
>>>> In our current context (as we have no courses in Sakai OAE) it  
>>>> would
>>>> actually make more sense to test this on a group, e.g. a "Calculus
>>>> Research Project group." This would likely also allow us to obtain
>>>> information about how users in groups function. Do you think it  
>>>> would be
>>>> problematic to make that change?
>>>
>>> In general, it would be good if we could keep the testing scenario
>>> as close to the original test as possible, just to make it easier
>>> to analyse results coming from different institutions. However, in
>>> this case, I think the change you'd like to make should be fine.  
>>> You'll
>>> still be testing the same functionality and interface elements, as
>>> you're basically just changing the name and type.
>>>
>>> It's probably worth keeping in mind that the server will already  
>>> have
>>> the Maths course pre-created for you. This means that you'll have
>>> to create the Mathematics Research Project group yourself and modify
>>> the content of the messages present in the system. It is important
>>> to keep the content of the group and messages as close as possible
>>> to the originals.
>>
>> We understand that there will be extra work involved if we make  
>> this change, and will will weigh this when deciding whether or not  
>> to do it. We definitely agree that the goals should be the same.
>>
>>>
>>>> Additionally, there are a few other things we'd like to add to  
>>>> the test to
>>>> investigate the usability of other things that are important to  
>>>> us at
>>>> Berkeley (e.g. categories, profile). We are hoping information  
>>>> from some
>>>> additional tasks we add would still be helpful to the managed  
>>>> project?
>>>
>>> I think it's great that you're keen to test other things as well,  
>>> but
>>> for this first package I would encourage institutions to run the  
>>> test
>>> as closely as possible to the original without changing or expanding
>>> the scope and goals. We're still trying to figure out how we can
>>> best analyse and summarise results coming from different  
>>> institutions,
>>> so for now it would be more helpful if all participating  
>>> institutions
>>> conducted the same focused test.
>>
>> We agree that analysis will be difficult. We had talked with Nico  
>> about the new OAE UX group (of which we hope you will be a  
>> part! :)) working on this. Are you envisioning the Sakai OAE UX  
>> group analyzing the results in some way together, or that you would  
>> take the lead on analysis and the rest of us would assist you in  
>> some fashion? Perhaps we can talk sometime soon about how we can  
>> help. We have been meeting (irregularly so far) Tuesdays at 9am PDT  
>> if that happens to work for you, or we'd be happy to set up a time  
>> anytime this week.
>>
>>>
>>> Another reason why it might not be the best time to test  
>>> categories and
>>> profiles is that both of these are currently undergoing design  
>>> changes.
>>> It feels more useful to create a usability package that tests these
>>> changes, rather than the current solution which will disappear soon.
>>
>> Great, that's exactly the sort of feedback we were looking for. Not  
>> testing these things makes sense if the changes aren't out by the  
>> time we do our tests. Any idea when these changes will be made?
>>
>>>
>>> However, it's really valuable to find out what you would like to  
>>> test
>>> as well. The goal is to send out small focused usability test  
>>> packages
>>> at regular times, and that information could help us in deciding  
>>> what
>>> to test next. Tomorrow, I will create a Confluence page where  
>>> everyone
>>> can list what they would like to test in the next usability  
>>> package. I'll
>>> send out a note once that page is up.
>>
>> It would be great to discuss strategy for future usability testing  
>> sessions together--and we think it's a fantastic idea to create a  
>> confluence page to discuss what to test next!  Since finding &  
>> scheduling users takes us quite a bit of time (as we endeavor to  
>> get a realistic sample as opposed to doing 'friends & family (or  
>> colleague)' testing), it would be easier for us to have fewer  
>> testing sessions which cover more ground. I'd be interested to hear  
>> about how other institutions who plan to participate in the testing  
>> sessions feel.
>>
>> Cheers,
>> Allison
>>
>>>
>>> Best wishes,
>>> Anne-Sophie
>>>
>>
>> Allison Bloodworth
>> Senior User Experience Designer
>> Educational Technology Services
>> University of California, Berkeley
>> [hidden email]
>> 510-289-6627
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> sakai-ui-dev mailing list
>> [hidden email]
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev
>
> Allison Bloodworth
> Senior User Experience Designer
> Educational Technology Services
> University of California, Berkeley
> [hidden email]
> 510-289-6627
>
>
>
>
>
>
>
> _______________________________________________
> oae-urg mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/oae-urg
> _______________________________________________
> sakai-ux mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>
> TO UNSUBSCRIBE: send email to [hidden email]
>  with a subject of "unsubscribe"

Daphne Ogle-Glenn
Senior Interaction Designer
University of California, Berkeley
Educational Technology Services
[hidden email]
cell (925)348-4372





_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Anne-Sophie De Baets Anne-Sophie De Baets
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Allison Bloodworth
Hi Allison,


Thanks for all the feedback!
I've finally found some time to add a note about 'clearing the changes
that participants made' between sessions. You can now find this extra
information in '2. Detailed Manual' under III.5, and I've also added it in
the intro on '13. Expected Outcomes'.  It's just a small note to make
people aware of it so nothing detailed of what specific changes have to be
cleared as this also depends on what participants do during the test (this
can vary a lot).

As these changes have now been made, I've also uploaded the newest version
of the ZIP on the confluence page, so institutions that plan to do the
testing in the future, should use this ZIP as well.

About changing the permissions on the test servers: don't worry about
making these notes because I could as well change these permissions to the
test servers instead.
As soon as Nico sets up another server, I'll make the changes to these
permissions.


Many thanks,
Anne-Sophie


> Hi folks,
>
> At UC Berkeley we've completed 4 usability tests and have 6 more scheduled
> by the end of next Tuesday. We decided not to modify the protocol by
> changing it to a project site or adding additional tasks.
>
> We have learned some things about the test environment --including a
> change that is needed to the permissions in the environment before
> starting the test, and some cleanup of the environment that is necessary
> between tests because the same "Elementary Calculus" course is being used
> by each tester.
>
> Is anyone else doing usability testing? If so I'll write up what we've
> learned about these things in detail.
>
> Cheers,
> Allison
>
> On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:
>
>> Hi Anne-Sophie,
>>
>> Thanks much for your responses and patience with us as Rachel, Daphne &
>> I came up with this feedback after taking a closer look at the test
>> together. :) Comments below...
>>
>> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
>>
>>> Hi Allison,
>>>
>>> Thanks a lot for your feedback. Some comments inline:
>>>
>>>> Rachel & I are going over the protocol in detail now and it would be
>>>> very
>>>> helpful to actually run through it on a test server. However, we don't
>>>> want to start Berkeley's countdown clock on having our server for 3
>>>> weeks
>>>> quite yet--is there a server that is set up now that we could use to
>>>> run
>>>> through the test?
>>>
>>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>>> server, using the same accounts and data as described in the manual,
>>> which have been created on there as well. However, the server is not
>>> running the latest code, so bugs might still exist. We'll reset the
>>> server to its initial state from time to time, meaning that you will
>>> lose all of the changes you make to the test accounts and data.
>>
>> Great, thanks! We took a quick look here and the server is a bit slow
>> but this should help us get an idea before we get our own server.
>>
>>>
>>>> In our current context (as we have no courses in Sakai OAE) it would
>>>> actually make more sense to test this on a group, e.g. a "Calculus
>>>> Research Project group." This would likely also allow us to obtain
>>>> information about how users in groups function. Do you think it would
>>>> be
>>>> problematic to make that change?
>>>
>>> In general, it would be good if we could keep the testing scenario
>>> as close to the original test as possible, just to make it easier
>>> to analyse results coming from different institutions. However, in
>>> this case, I think the change you'd like to make should be fine. You'll
>>> still be testing the same functionality and interface elements, as
>>> you're basically just changing the name and type.
>>>
>>> It's probably worth keeping in mind that the server will already have
>>> the Maths course pre-created for you. This means that you'll have
>>> to create the Mathematics Research Project group yourself and modify
>>> the content of the messages present in the system. It is important
>>> to keep the content of the group and messages as close as possible
>>> to the originals.
>>
>> We understand that there will be extra work involved if we make this
>> change, and will will weigh this when deciding whether or not to do it.
>> We definitely agree that the goals should be the same.
>>
>>>
>>>> Additionally, there are a few other things we'd like to add to the
>>>> test to
>>>> investigate the usability of other things that are important to us at
>>>> Berkeley (e.g. categories, profile). We are hoping information from
>>>> some
>>>> additional tasks we add would still be helpful to the managed project?
>>>
>>> I think it's great that you're keen to test other things as well, but
>>> for this first package I would encourage institutions to run the test
>>> as closely as possible to the original without changing or expanding
>>> the scope and goals. We're still trying to figure out how we can
>>> best analyse and summarise results coming from different institutions,
>>> so for now it would be more helpful if all participating institutions
>>> conducted the same focused test.
>>
>> We agree that analysis will be difficult. We had talked with Nico about
>> the new OAE UX group (of which we hope you will be a part! :)) working
>> on this. Are you envisioning the Sakai OAE UX group analyzing the
>> results in some way together, or that you would take the lead on
>> analysis and the rest of us would assist you in some fashion? Perhaps we
>> can talk sometime soon about how we can help. We have been meeting
>> (irregularly so far) Tuesdays at 9am PDT if that happens to work for
>> you, or we'd be happy to set up a time anytime this week.
>>
>>>
>>> Another reason why it might not be the best time to test categories and
>>> profiles is that both of these are currently undergoing design changes.
>>> It feels more useful to create a usability package that tests these
>>> changes, rather than the current solution which will disappear soon.
>>
>> Great, that's exactly the sort of feedback we were looking for. Not
>> testing these things makes sense if the changes aren't out by the time
>> we do our tests. Any idea when these changes will be made?
>>
>>>
>>> However, it's really valuable to find out what you would like to test
>>> as well. The goal is to send out small focused usability test packages
>>> at regular times, and that information could help us in deciding what
>>> to test next. Tomorrow, I will create a Confluence page where everyone
>>> can list what they would like to test in the next usability package.
>>> I'll
>>> send out a note once that page is up.
>>
>> It would be great to discuss strategy for future usability testing
>> sessions together--and we think it's a fantastic idea to create a
>> confluence page to discuss what to test next!  Since finding &
>> scheduling users takes us quite a bit of time (as we endeavor to get a
>> realistic sample as opposed to doing 'friends & family (or colleague)'
>> testing), it would be easier for us to have fewer testing sessions which
>> cover more ground. I'd be interested to hear about how other
>> institutions who plan to participate in the testing sessions feel.
>>
>> Cheers,
>> Allison
>>
>>>
>>> Best wishes,
>>> Anne-Sophie
>>>
>>
>> Allison Bloodworth
>> Senior User Experience Designer
>> Educational Technology Services
>> University of California, Berkeley
>> [hidden email]
>> 510-289-6627
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> sakai-ui-dev mailing list
>> [hidden email]
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev
>
> Allison Bloodworth
> Senior User Experience Designer
> Educational Technology Services
> University of California, Berkeley
> [hidden email]
> 510-289-6627
>
>
>
>
>
>
>
>


_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"
Anne-Sophie De Baets Anne-Sophie De Baets
Reply | Threaded
Open this post in threaded view
|

Re: [DG: User Experience] [oae-urg] [Building Sakai] Sakai OAE Usability Testing Package 1

In reply to this post by Daphne Ogle
Hi Daphne,


I'm not really that familiar with Confluence to be able to say in what way
we could make things private (for example for the raw notes and videos).
I agree that it's probably okay to put the profiles on Confluence without
special privacy settings, as the profiles don't mention full names and
because no voice or face will be recognised through that.
However, I don't think it's beneficial for the tests if people already
look at these profiles before they've carried out their own tests, as they
might get prejudices before starting their own tests.
So yes I don't mind putting them on confluence, but only if people looking
at it, are people who don't intend to carry out tests themselves, or if
they've already finished their own tests.

Anne-Sophie


> Hi Keli,
>
> We'll be sharing our results back to Anne-Sophie in the form of the
> profiles she's asked along with our raw notes and video.  For privacy
> reasons, the raw notes and video should be kept private within the
> project team (do we have a way to post these things and restrict
> access?) but my assumption is Anne-Sophie will make all the profiles
> public.  We should have those available next week.
>
> -Daphne
>
> On Sep 9, 2011, at 10:16 AM, Keli Sato Amann wrote:
>
>> We have not, but we'd love to hear about your results.
>>
>> ----- Original Message -----
>> From: "Allison Bloodworth" <[hidden email]>
>> To: "Sakai UI Development" <[hidden email]>
>> Cc: "Anne-Sophie De Baets" <[hidden email]>, "Sakai OAE User
>> Reference Group" <[hidden email]>, "Sakai UX"
>> <[hidden email]
>> >, "Sakai Developers" <[hidden email]>
>> Sent: Friday, September 9, 2011 9:27:54 AM
>> Subject: Re: [oae-urg] [Building Sakai] Sakai OAE Usability Testing
>> Package 1
>>
>> Hi folks,
>>
>> At UC Berkeley we've completed 4 usability tests and have 6 more
>> scheduled by the end of next Tuesday. We decided not to modify the
>> protocol by changing it to a project site or adding additional tasks.
>>
>> We have learned some things about the test environment --including a
>> change that is needed to the permissions in the environment before
>> starting the test, and some cleanup of the environment that is
>> necessary between tests because the same "Elementary Calculus"
>> course is being used by each tester.
>>
>> Is anyone else doing usability testing? If so I'll write up what
>> we've learned about these things in detail.
>>
>> Cheers,
>> Allison
>>
>> On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:
>>
>>> Hi Anne-Sophie,
>>>
>>> Thanks much for your responses and patience with us as Rachel,
>>> Daphne & I came up with this feedback after taking a closer look at
>>> the test together. :) Comments below...
>>>
>>> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
>>>
>>>> Hi Allison,
>>>>
>>>> Thanks a lot for your feedback. Some comments inline:
>>>>
>>>>> Rachel & I are going over the protocol in detail now and it would
>>>>> be very
>>>>> helpful to actually run through it on a test server. However, we
>>>>> don't
>>>>> want to start Berkeley's countdown clock on having our server for
>>>>> 3 weeks
>>>>> quite yet--is there a server that is set up now that we could use
>>>>> to run
>>>>> through the test?
>>>>
>>>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>>>> server, using the same accounts and data as described in the manual,
>>>> which have been created on there as well. However, the server is not
>>>> running the latest code, so bugs might still exist. We'll reset the
>>>> server to its initial state from time to time, meaning that you will
>>>> lose all of the changes you make to the test accounts and data.
>>>
>>> Great, thanks! We took a quick look here and the server is a bit
>>> slow but this should help us get an idea before we get our own
>>> server.
>>>
>>>>
>>>>> In our current context (as we have no courses in Sakai OAE) it
>>>>> would
>>>>> actually make more sense to test this on a group, e.g. a "Calculus
>>>>> Research Project group." This would likely also allow us to obtain
>>>>> information about how users in groups function. Do you think it
>>>>> would be
>>>>> problematic to make that change?
>>>>
>>>> In general, it would be good if we could keep the testing scenario
>>>> as close to the original test as possible, just to make it easier
>>>> to analyse results coming from different institutions. However, in
>>>> this case, I think the change you'd like to make should be fine.
>>>> You'll
>>>> still be testing the same functionality and interface elements, as
>>>> you're basically just changing the name and type.
>>>>
>>>> It's probably worth keeping in mind that the server will already
>>>> have
>>>> the Maths course pre-created for you. This means that you'll have
>>>> to create the Mathematics Research Project group yourself and modify
>>>> the content of the messages present in the system. It is important
>>>> to keep the content of the group and messages as close as possible
>>>> to the originals.
>>>
>>> We understand that there will be extra work involved if we make
>>> this change, and will will weigh this when deciding whether or not
>>> to do it. We definitely agree that the goals should be the same.
>>>
>>>>
>>>>> Additionally, there are a few other things we'd like to add to
>>>>> the test to
>>>>> investigate the usability of other things that are important to
>>>>> us at
>>>>> Berkeley (e.g. categories, profile). We are hoping information
>>>>> from some
>>>>> additional tasks we add would still be helpful to the managed
>>>>> project?
>>>>
>>>> I think it's great that you're keen to test other things as well,
>>>> but
>>>> for this first package I would encourage institutions to run the
>>>> test
>>>> as closely as possible to the original without changing or expanding
>>>> the scope and goals. We're still trying to figure out how we can
>>>> best analyse and summarise results coming from different
>>>> institutions,
>>>> so for now it would be more helpful if all participating
>>>> institutions
>>>> conducted the same focused test.
>>>
>>> We agree that analysis will be difficult. We had talked with Nico
>>> about the new OAE UX group (of which we hope you will be a
>>> part! :)) working on this. Are you envisioning the Sakai OAE UX
>>> group analyzing the results in some way together, or that you would
>>> take the lead on analysis and the rest of us would assist you in
>>> some fashion? Perhaps we can talk sometime soon about how we can
>>> help. We have been meeting (irregularly so far) Tuesdays at 9am PDT
>>> if that happens to work for you, or we'd be happy to set up a time
>>> anytime this week.
>>>
>>>>
>>>> Another reason why it might not be the best time to test
>>>> categories and
>>>> profiles is that both of these are currently undergoing design
>>>> changes.
>>>> It feels more useful to create a usability package that tests these
>>>> changes, rather than the current solution which will disappear soon.
>>>
>>> Great, that's exactly the sort of feedback we were looking for. Not
>>> testing these things makes sense if the changes aren't out by the
>>> time we do our tests. Any idea when these changes will be made?
>>>
>>>>
>>>> However, it's really valuable to find out what you would like to
>>>> test
>>>> as well. The goal is to send out small focused usability test
>>>> packages
>>>> at regular times, and that information could help us in deciding
>>>> what
>>>> to test next. Tomorrow, I will create a Confluence page where
>>>> everyone
>>>> can list what they would like to test in the next usability
>>>> package. I'll
>>>> send out a note once that page is up.
>>>
>>> It would be great to discuss strategy for future usability testing
>>> sessions together--and we think it's a fantastic idea to create a
>>> confluence page to discuss what to test next!  Since finding &
>>> scheduling users takes us quite a bit of time (as we endeavor to
>>> get a realistic sample as opposed to doing 'friends & family (or
>>> colleague)' testing), it would be easier for us to have fewer
>>> testing sessions which cover more ground. I'd be interested to hear
>>> about how other institutions who plan to participate in the testing
>>> sessions feel.
>>>
>>> Cheers,
>>> Allison
>>>
>>>>
>>>> Best wishes,
>>>> Anne-Sophie
>>>>
>>>
>>> Allison Bloodworth
>>> Senior User Experience Designer
>>> Educational Technology Services
>>> University of California, Berkeley
>>> [hidden email]
>>> 510-289-6627
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> sakai-ui-dev mailing list
>>> [hidden email]
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev
>>
>> Allison Bloodworth
>> Senior User Experience Designer
>> Educational Technology Services
>> University of California, Berkeley
>> [hidden email]
>> 510-289-6627
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> oae-urg mailing list
>> [hidden email]
>> http://collab.sakaiproject.org/mailman/listinfo/oae-urg
>> _______________________________________________
>> sakai-ux mailing list
>> [hidden email]
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>
>> TO UNSUBSCRIBE: send email to
>> [hidden email]
>>  with a subject of "unsubscribe"
>
> Daphne Ogle-Glenn
> Senior Interaction Designer
> University of California, Berkeley
> Educational Technology Services
> [hidden email]
> cell (925)348-4372
>
>
>
>
>
> _______________________________________________
> sakai-ux mailing list
> [hidden email]
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>
> TO UNSUBSCRIBE: send email to [hidden email]
> with a subject of "unsubscribe"
>


_______________________________________________
sakai-ux mailing list
[hidden email]
http://collab.sakaiproject.org/mailman/listinfo/sakai-ux

TO UNSUBSCRIBE: send email to [hidden email] with a subject of "unsubscribe"